Skip to main content

Chroma

Availability
Available CoreNot available StandardNot available PlusNot available ProNot available Enterprise FlexAvailable PyAirbyte
Support Level
Marketplace
Connector Version
0.0.54 (last updated a year ago)
CDK Version
0.81.6
Sync Success Rate
Low
Usage Rate
Low
Definition ID
0b75218b-f702-4a28-85ac-34d3d84c0fc2

This page guides you through setting up the Chroma destination connector.

The connector splits the text fields of your records into chunks, turns each chunk into an embedding vector, and writes the vectors, the chunk text, and the record metadata to a single Chroma collection. You can either let Airbyte compute the embeddings with an external service, pass embeddings that already exist in your records, or let Chroma compute them with its own default embedding function.

Prerequisites​

  • A Chroma database. The connector can connect to a Chroma server over HTTP, or read and write a database directory on the machine running Airbyte.
  • A username and password, if your Chroma server requires basic authentication.
  • An API key for OpenAI, Azure OpenAI, Cohere, or another OpenAI-compatible embedding service, if you want Airbyte to compute the embeddings. You don't need one if your records already contain embeddings or if you use Chroma's default embedding function.

Supported sync modes​

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedYes

Overwrite syncs don't delete the collection. At the start of the sync, the connector deletes the documents whose _ab_stream metadata value matches the streams being overwritten, so documents from other streams and from other tools stay in place.

Deduplicating syncs require a primary key on the stream. Before writing a record, the connector deletes the documents whose _ab_record_id metadata value matches that record's primary key.

Output schema​

All source streams are written to one Chroma collection, which you name in the connector configuration. Each chunk becomes one document in that collection:

  • The document ID is a randomly generated UUID. Source primary keys are stored in metadata instead, so you can't look a record up by its source ID.
  • The chunk text is stored as the document, unless you enable Do not store raw text.
  • The embedding is stored as the document's embedding. If you use Chroma's default embedding function, Airbyte sends no embedding and Chroma computes one.
  • The fields you select as metadata are stored as document metadata, along with _ab_stream (the stream identifier, in the form namespace_stream when the stream has a namespace) and _ab_record_id (the record's primary key, for deduplicating streams).

Chroma metadata values must be strings, numbers, or booleans. The connector JSON-encodes any other value, such as an object or an array, into a string.

Namespace support​

Source namespaces don't create separate Chroma collections. Everything is written to the collection you configure, and the namespace becomes part of the _ab_stream metadata value, which you can filter on when you query Chroma.

Set up the Chroma destination​

Connection mode​

Choose one of the following connection modes.

Client/Server Mode connects to a running Chroma server:

  • Host: the hostname of the Chroma instance, for example localhost.
  • Port: the port the Chroma instance listens on, for example 8000.
  • SSL: whether to connect over HTTPS.
  • Username and Password: only needed if the server is configured for basic authentication. Leave both empty otherwise.

Make sure the Chroma server is reachable from Airbyte. If it runs inside a VPC, allow access from the IP address that Airbyte syncs from.

Persistent Client Mode stores the database in a directory on the machine running Airbyte:

  • Path: an absolute path prefixed with /local, for example /local/chroma.

In Docker deployments, /local is mapped to the directory Airbyte mounts for local files, which is /tmp/airbyte_local unless you change the LOCAL_ROOT environment variable. On macOS, Docker Desktop must be allowed to share /tmp and /private, because /tmp is a symlink to /private/tmp. Grant access in Settings > Resources > File sharing, then apply and restart.

danger

Persistent Client Mode isn't supported on Kubernetes deployments. Use Client/Server Mode instead.

Collection name​

Set Collection Name to the collection you want to load data into. The connector creates the collection if it doesn't exist. Chroma uses the name in URLs, so it must:

  • Be between 3 and 63 characters long.
  • Start and end with a lowercase letter or a digit.
  • Contain only alphanumeric characters, dots, dashes, and underscores.
  • Contain no two consecutive dots.
  • Not be a valid IPv4 address.

The connector checks these rules during the connection test and reports a specific error when the name is invalid.

Embedding​

Pick how the embeddings are produced:

  • OpenAI: uses text-embedding-ada-002 with 1536 dimensions. Requires an OpenAI API key.
  • Azure OpenAI: uses text-embedding-ada-002 with 1536 dimensions from your own Azure resource. Requires the Azure OpenAI API key, Resource base URL, and Deployment.
  • Cohere: requires a Cohere API key.
  • OpenAI-compatible: for self-hosted or third-party services that expose the OpenAI embeddings API. Requires the Base URL, Embedding dimensions, and, depending on the service, an API key and Model name.
  • From Field: uses an embedding that already exists in the record. Set Field name to the field holding the vector and Embedding dimensions to its length. Records whose vector has a different length fail the sync.
  • Fake: random vectors with 1536 dimensions, for testing a pipeline without paying for embeddings.
  • Chroma Default Embedding Function: Airbyte sends no embedding, and Chroma embeds the documents itself with its default embedding function, which runs the all-MiniLM-L6-v2 model on the machine running Chroma. No API key is needed.

Processing​

These options control how records become chunks:

  • Chunk size (required): the maximum size of a chunk in tokens, up to 8191. Keep it small enough for the context window of the model you query with.
  • Chunk overlap: how many tokens consecutive chunks share, which helps preserve context across chunk boundaries. Defaults to 0.
  • Text fields to embed: the record fields to embed. Use dot notation for nested fields, such as user.name, and wildcards for arrays, such as users.*.name. If you leave this empty, all fields are embedded.
  • Fields to store as metadata: the record fields to store as document metadata. If you leave this empty, all fields are stored as metadata.
  • Text splitter: how to split text that exceeds the chunk size. You can split by separator, by Markdown header level, or by the syntax of a programming language.
  • Field name mappings: renames source fields before they're written.
  • Do not store raw text: stores only the embedding and metadata. Use this when the source text is sensitive or already available elsewhere. Retrieval-augmented generation workflows that read the chunk text from Chroma break when you enable it.

The connector writes documents to Chroma in batches of 128 chunks.

Reference​

Config fields reference

Field
Type
Property name
object
embedding
object
indexing
object
processing
boolean
omit_raw_text

Changelog​

Expand to review
VersionDatePull RequestSubject
0.0.552026-08-1384361Update the CDK to remediate CVE-2025-68664 in the langchain dependency
0.0.542025-05-0359326Update dependencies
0.0.532025-04-2658256Update dependencies
0.0.522025-04-1257652Update dependencies
0.0.512025-04-0557169Update dependencies
0.0.502025-03-2956624Update dependencies
0.0.492025-03-2256120Update dependencies
0.0.482025-03-0855393Update dependencies
0.0.472025-03-0154852Update dependencies
0.0.462025-02-2254209Update dependencies
0.0.452025-02-1553930Update dependencies
0.0.442025-02-0853428Update dependencies
0.0.432025-02-0152941Update dependencies
0.0.422025-01-2552189Update dependencies
0.0.412025-01-1851744Update dependencies
0.0.402025-01-1151296Update dependencies
0.0.392025-01-0450913Update dependencies
0.0.382024-12-2850445Update dependencies
0.0.372024-12-2150221Update dependencies
0.0.362024-12-1448956Update dependencies
0.0.352024-11-2548668Update dependencies
0.0.342024-11-0448236Update dependencies
0.0.332024-10-2947053Update dependencies
0.0.322024-10-1246434Update dependencies
0.0.312024-09-2846192Update dependencies
0.0.302024-09-2145553Update dependencies
0.0.292024-09-0745322Update dependencies
0.0.282024-08-3145017Update dependencies
0.0.272024-08-2444717Update dependencies
0.0.262024-08-2244530Update test dependencies
0.0.252024-08-1744292Update dependencies
0.0.242024-08-1243734Update dependencies
0.0.232024-08-1043702Update dependencies
0.0.222024-08-0343133Update dependencies
0.0.212024-07-2742628Update dependencies
0.0.202024-07-2042160Update dependencies
0.0.192024-07-1341802Update dependencies
0.0.182024-07-1041384Update dependencies
0.0.172024-07-0941165Update dependencies
0.0.162024-07-0640926Update dependencies
0.0.152024-06-2940634Update dependencies
0.0.142024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.0.132024-06-2540431Update dependencies
0.0.122024-06-2340222Update dependencies
0.0.112024-06-2240068Update dependencies
0.0.102024-04-15#37333Updated CDK & pytest version to fix security vulnerabilities
0.0.92023-12-11#33303Fix bug with embedding special tokens
0.0.82023-12-01#32697Allow omitting raw text
0.0.72023-11-16#32608Support deleting records for CDC sources
0.0.62023-11-13#32357Improve spec schema
0.0.52023-10-23#31563Add field mapping option
0.0.42023-10-15#31329Add OpenAI-compatible embedder option
0.0.32023-10-04#31075Fix OpenAI embedder batch size
0.0.22023-09-29#30820Update CDK
0.0.12023-09-08#30023🎉 New Destination: Chroma (Vector Database)
Was this page helpful?