Go API Reference

Go API Reference v5.0.0-rc.10¶

Functions¶

ExtractBytes()¶

Extract content from a byte array.

This is the main entry point for in-memory extraction. It performs the following steps:

1. Validate MIME type
2. Handle legacy format conversion if needed
3. Select appropriate extractor from registry
4. Extract content
5. Run post-processing pipeline

Returns:

An ExtractionResult containing the extracted content and metadata.

Errors:

Returns KreuzbergError.Validation if MIME type is invalid. Returns KreuzbergError.UnsupportedFormat if MIME type is not supported.

Signature:

func ExtractBytes(content []byte, mimeType string, config ExtractionConfig) (ExtractionResult, error)

Parameters:

Returns: ExtractionResult Errors: Returns error.

ExtractFile()¶

Extract content from a file.

This is the main entry point for file-based extraction. It performs the following steps:

1. Check cache for existing result (if caching enabled)
2. Detect or validate MIME type
3. Select appropriate extractor from registry
4. Extract content
5. Run post-processing pipeline
6. Store result in cache (if caching enabled)

Returns:

An ExtractionResult containing the extracted content and metadata.

Errors:

Returns KreuzbergError.Io if the file doesn't exist (NotFound) or for other file I/O errors. Returns KreuzbergError.UnsupportedFormat if MIME type is not supported.

Signature:

func ExtractFile(path string, mimeType string, config ExtractionConfig) (ExtractionResult, error)

Parameters:

Returns: ExtractionResult Errors: Returns error.

ExtractFileSync()¶

Synchronous wrapper for extract_file.

This is a convenience function that blocks the current thread until extraction completes. For async code, use extract_file directly.

Uses the global Tokio runtime for 100x+ performance improvement over creating a new runtime per call. Always uses the global runtime to avoid nested runtime issues.

This function is only available with the tokio-runtime feature. For WASM targets, use a truly synchronous extraction approach instead.

Signature:

func ExtractFileSync(path string, mimeType string, config ExtractionConfig) (ExtractionResult, error)

Parameters:

Returns: ExtractionResult Errors: Returns error.

ExtractBytesSync()¶

Synchronous wrapper for extract_bytes.

Uses the global Tokio runtime for 100x+ performance improvement over creating a new runtime per call.

With the tokio-runtime feature, this blocks the current thread using the global Tokio runtime. Without it (WASM), this calls a truly synchronous implementation.

Signature:

func ExtractBytesSync(content []byte, mimeType string, config ExtractionConfig) (ExtractionResult, error)

Parameters:

Returns: ExtractionResult Errors: Returns error.

BatchExtractFilesSync()¶

Synchronous wrapper for batch_extract_files.

Uses the global Tokio runtime for optimal performance. Only available with tokio-runtime (WASM has no filesystem).

Signature:

func BatchExtractFilesSync(items []BatchFileItem, config ExtractionConfig) ([]ExtractionResult, error)

Parameters:

Returns: []ExtractionResult Errors: Returns error.

BatchExtractBytesSync()¶

Synchronous wrapper for batch_extract_bytes.

Uses the global Tokio runtime for optimal performance. With the tokio-runtime feature, this blocks the current thread using the global Tokio runtime. Without it (WASM), this calls a truly synchronous implementation that iterates through items and calls extract_bytes_sync().

Signature:

func BatchExtractBytesSync(items []BatchBytesItem, config ExtractionConfig) ([]ExtractionResult, error)

Parameters:

Returns: []ExtractionResult Errors: Returns error.

BatchExtractFiles()¶

Extract content from multiple files concurrently.

This function processes multiple files in parallel, automatically managing concurrency to prevent resource exhaustion. The concurrency limit can be configured via ExtractionConfig.max_concurrent_extractions or defaults to (num_cpus * 1.5).ceil().

Each file can optionally specify a FileExtractionConfig that overrides specific fields from the batch-level config. Pass nil for a file to use the batch defaults. Batch-level settings like max_concurrent_extractions and use_cache are always taken from the batch-level config.

per-file configuration overrides.

• config - Batch-level extraction configuration (provides defaults and batch settings)

Returns:

A vector of ExtractionResult in the same order as the input items.

Errors:

Individual file errors are captured in the result metadata. System errors (IO, RuntimeError equivalents) will bubble up and fail the entire batch.

Simple usage with no per-file overrides:

Per-file configuration overrides:

Signature:

func BatchExtractFiles(items []BatchFileItem, config ExtractionConfig) ([]ExtractionResult, error)

Parameters:

Returns: []ExtractionResult Errors: Returns error.

BatchExtractBytes()¶

Extract content from multiple byte arrays concurrently.

This function processes multiple byte arrays in parallel, automatically managing concurrency to prevent resource exhaustion. The concurrency limit can be configured via ExtractionConfig.max_concurrent_extractions or defaults to (num_cpus * 1.5).ceil().

Each item can optionally specify a FileExtractionConfig that overrides specific fields from the batch-level config. Pass nil as the config to use the batch-level defaults for that item.

MIME type, and optional per-item configuration overrides.

• config - Batch-level extraction configuration

Returns:

A vector of ExtractionResult in the same order as the input items.

Simple usage with no per-item overrides:

Per-item configuration overrides:

Signature:

func BatchExtractBytes(items []BatchBytesItem, config ExtractionConfig) ([]ExtractionResult, error)

Parameters:

Returns: []ExtractionResult Errors: Returns error.

DetectMimeTypeFromBytes()¶

Detect MIME type from raw file bytes.

Uses magic byte signatures to detect file type from content. Falls back to infer crate for comprehensive detection.

For ZIP-based files, inspects contents to distinguish Office Open XML formats (DOCX, XLSX, PPTX) from plain ZIP archives.

Returns:

The detected MIME type string.

Errors:

Returns KreuzbergError.UnsupportedFormat if MIME type cannot be determined.

Signature:

func DetectMimeTypeFromBytes(content []byte) (string, error)

Parameters:

Returns: string Errors: Returns error.

GetExtensionsForMime()¶

Get file extensions for a given MIME type.

Returns all known file extensions that map to the specified MIME type.

Returns:

A vector of file extensions (without leading dot) for the MIME type.

Signature:

func GetExtensionsForMime(mimeType string) ([]string, error)

Parameters:

Returns: []string Errors: Returns error.

DetectQrCodes()¶

Detect QR codes in the bytes of an ExtractedImage.

format_hint is currently unused — the image crate auto-detects the container format from magic bytes — but the parameter is retained so future backends (e.g. a WebP-via-webp-decoder variant) can use it without an API break.

Returns an empty vector on any of:

• Empty input.
• Image-decode failure.
• No QR grids detected.
• All detected grids fail to decode.

Successfully decoded QR codes carry their payload, a confidence of 1.0 (rqrr does not expose per-grid confidence; a successful decode is treated as high-confidence by convention), and the pixel-space bounding box derived from the four corner points of the grid.

Signature:

func DetectQrCodes(imageBytes []byte, formatHint string) []QrCode

Parameters:

Returns: []QrCode

ClearEmbeddingBackends()¶

Clear all embedding backends from the global registry.

Calls shutdown() on every registered backend, then empties the registry.

Errors:

• Any error returned by a backend's shutdown() method. The first error encountered stops processing of remaining backends.

Signature:

func ClearEmbeddingBackends() error

Returns: ` **Errors:** Returnserror`.

ListEmbeddingBackends()¶

List the names of all registered embedding backends.

Used by kreuzberg-cli, the api/mcp endpoints, and generated language bindings.

Signature:

func ListEmbeddingBackends() ([]string, error)

Returns: []string Errors: Returns error.

ListDocumentExtractors()¶

List names of all registered document extractors.

Signature:

func ListDocumentExtractors() ([]string, error)

Returns: []string Errors: Returns error.

ClearDocumentExtractors()¶

Clear all document extractors from the global registry.

Calls shutdown() on every registered extractor, then empties the registry.

Errors:

• Any error returned by an extractor's shutdown() method. The first error encountered stops processing of remaining extractors.

Signature:

func ClearDocumentExtractors() error

Returns: ` **Errors:** Returnserror`.

ListOcrBackends()¶

List all registered OCR backends.

Returns the names of all OCR backends currently registered in the global registry.

Returns:

A vector of OCR backend names.

Signature:

func ListOcrBackends() ([]string, error)

Returns: []string Errors: Returns error.

ClearOcrBackends()¶

Clear all OCR backends from the global registry.

Removes all OCR backends and calls their shutdown() methods.

Returns:

• Ok(()) if all backends were cleared successfully
• Err(...) if any shutdown method failed

Signature:

func ClearOcrBackends() error

Returns: ` **Errors:** Returnserror`.

RegisterBuiltin()¶

Register every built-in post-processor enabled by the active feature set.

This is the single entry point that callers (including register_default_post_processors) use to populate the global post-processor registry with the in-tree built-ins. Each submodule's own register function is gated by its feature flag so this aggregate stays safe to call on any target.

Signature:

func RegisterBuiltin() error

Returns: ` **Errors:** Returnserror`.

ListPostProcessors()¶

List all registered post-processor names.

Returns a vector of all post-processor names currently registered in the global registry.

Returns:

• Ok(Vec<String>) - Vector of post-processor names
• Err(...) if the registry lock is poisoned

Signature:

func ListPostProcessors() ([]string, error)

Returns: []string Errors: Returns error.

ClearPostProcessors()¶

Remove all registered post-processors.

Signature:

func ClearPostProcessors() error

Returns: ` **Errors:** Returnserror`.

ListRenderers()¶

List names of all registered renderers.

Errors:

Returns an error if the registry lock is poisoned.

Signature:

func ListRenderers() ([]string, error)

Returns: []string Errors: Returns error.

ClearRenderers()¶

Clear all renderers from the global registry.

Removes every renderer, including the built-in defaults (markdown, html, djot, plain). After calling this no renderers are registered; re-register as needed.

Errors:

Returns an error if the registry lock is poisoned.

Signature:

func ClearRenderers() error

Returns: ` **Errors:** Returnserror`.

ListValidators()¶

List names of all registered validators.

Signature:

func ListValidators() ([]string, error)

Returns: []string Errors: Returns error.

ClearValidators()¶

Remove all registered validators.

Signature:

func ClearValidators() error

Returns: ` **Errors:** Returnserror`.

ClassifyPages()¶

Run page classification against an extraction result.

Mutates result.page_classifications with one entry per non-empty page and appends every LLM call's usage to result.llm_usage.

Errors:

Returns the first error encountered when rendering the prompt or calling the LLM. Partially produced classifications are discarded so callers do not see a half-populated vector.

Signature:

func ClassifyPages(result ExtractionResult, config PageClassificationConfig) error

Parameters:

Returns: ` **Errors:** Returnserror`.

DownloadModel()¶

Eagerly download a NER model into the kreuzberg cache.

name is a HuggingFace repo id (e.g. urchade/gliner_multi-v2.1). The CLI flag kreuzberg warm --ner delegates here.

Signature:

func DownloadModel(name string, cacheDir string) (string, error)

Parameters:

Returns: string Errors: Returns error.

DefaultModelName()¶

Pinned default NER model identifier.

Signature:

func DefaultModelName() string

Returns: string

KnownModels()¶

All NER models kreuzberg knows about (used by --all-ner-models).

Signature:

func KnownModels() []string

Returns: []string

Redact()¶

Run pattern redaction (and optional NER-driven redaction) over result and rewrite every textual field. Populates result.redaction_report.

Signature:

func Redact(result ExtractionResult, config RedactionConfig) error

Parameters:

Returns: ` **Errors:** Returnserror`.

Summarize()¶

Score and return the top-N sentences from text, joined in original order.

language is an ISO 639 (or locale) code used to pick a stopword list; pass nil (or an unknown code) to fall back to English. max_tokens bounds the summary length by whitespace-separated tokens; nil falls back to DEFAULT_MAX_TOKENS.

Signature:

func Summarize(text string, language string, maxTokens uint32) *string

Parameters:

Returns: *string

TokenCount()¶

Count whitespace-separated tokens (used for token-budget bookkeeping by callers).

Signature:

func TokenCount(text string) uint32

Parameters:

Returns: uint32

TranslateResult()¶

Translate the extraction result in place.

Populates result.translation with the translated content, optionally the translated formatted_content (when preserve_markup = true), and rewrites every chunk's content field. Every LLM call's usage is appended to result.llm_usage.

Signature:

func TranslateResult(result ExtractionResult, config TranslationConfig) error

Parameters:

Returns: ` **Errors:** Returnserror`.

Compare()¶

Compare two extraction results and return a structured diff.

The comparison is purely structural — no I/O, no side effects. All fields of ExtractionDiff are populated according to the provided DiffOptions.

Signature:

func Compare(a ExtractionResult, b ExtractionResult, opts DiffOptions) ExtractionDiff

Parameters:

Returns: ExtractionDiff

ExtractRegionWithVlm()¶

Extract content from a pre-cropped image region using a VLM.

The caller is responsible for cropping the page image to the region's bounding box before calling this function. The image_bytes parameter must contain the raw bytes of the cropped region image (JPEG, PNG, WebP, etc.).

Returns:

Extracted Markdown text from the VLM, or an error if the VLM call fails.

Errors:

• Ocr if the VLM call fails or returns no content.
• MissingDependency if the liter-llm client cannot be initialised.

Signature:

func ExtractRegionWithVlm(imageBytes []byte, imageMime string, regionKind RegionKind, llmConfig LlmConfig, customPrompt string) (string, error)

Parameters:

Returns: string Errors: Returns error.

RenderPdfPageToPng()¶

Render a single PDF page to PNG bytes.

Returns raw PNG-encoded bytes for the specified page at the given DPI. Uses pdf_oxide with tiny-skia for pure-Rust rendering.

For pages with extreme dimensions (very wide vector diagrams, etc.) the effective DPI may be automatically reduced to avoid rasterizer failure. A warning is logged when this happens.

Errors:

Returns KreuzbergError.Parsing if the PDF cannot be opened, authenticated, or rendered, or if page_index is out of range.

Signature:

func RenderPdfPageToPng(pdfBytes []byte, pageIndex int, dpi int32, password string) ([]byte, error)

Parameters:

Returns: []byte Errors: Returns error.

DetectMimeType()¶

Detect the MIME type of a file at the given path.

Uses the file extension and optionally the file content to determine the MIME type. Set check_exists to true to verify the file exists before detection.

Signature:

func DetectMimeType(path string, checkExists bool) (string, error)

Parameters:

Returns: string Errors: Returns error.

EmbedTextsAsync()¶

Signature:

func EmbedTextsAsync(texts []string, config EmbeddingConfig) ([][]float32, error)

Parameters:

Returns: [][]float32 Errors: Returns error.

GetEmbeddingPreset()¶

Get an embedding preset by name.

Returns nil if no preset with the given name exists. Returns an owned clone so the value is safe to pass across FFI boundaries.

Signature:

func GetEmbeddingPreset(name string) *EmbeddingPreset

Parameters:

Returns: *EmbeddingPreset

ListEmbeddingPresets()¶

List the names of all available embedding presets.

Returns owned Strings so the values are safe to pass across FFI boundaries.

Signature:

func ListEmbeddingPresets() []string

Returns: []string

Types¶

AccelerationConfig¶

Hardware acceleration configuration for ONNX Runtime models.

Controls which execution provider (CPU, CoreML, CUDA, TensorRT) is used for inference in layout detection and embedding generation.

ArchiveEntry¶

A single file extracted from an archive.

When archives (ZIP, TAR, 7Z, GZIP) are extracted with recursive extraction enabled, each processable file produces its own full ExtractionResult.

ArchiveMetadata¶

Archive (ZIP/TAR/7Z) metadata.

Extracted from compressed archive files containing file lists and size information.

AudioMetadata¶

Audio/video file metadata.

Populated from container tags (ID3v2, MP4 atoms, Vorbis comments, etc.) and PCM decode properties. Available when the transcription-types feature is enabled.

BBox¶

Bounding box in original image coordinates (x1, y1) top-left, (x2, y2) bottom-right.

BatchBytesItem¶

Batch item for byte array extraction.

Used with batch_extract_bytes and batch_extract_bytes_sync to represent a single item in a batch extraction job.

BatchFileItem¶

Batch item for file extraction.

Used with batch_extract_files and batch_extract_files_sync to represent a single file in a batch extraction job.

BibtexMetadata¶

BibTeX bibliography metadata.

BoundingBox¶

Bounding box coordinates for element positioning.

CacheStats¶

Aggregate statistics for a kreuzberg cache directory.

CaptioningConfig¶

Configuration for the VLM captioning post-processor.

CellChange¶

A single changed cell within a table.

Defined here (rather than only in crate.diff) so RevisionDelta can reference it unconditionally, without requiring the diff Cargo feature. crate.diff re-exports this type verbatim.

Chunk¶

A text chunk with optional embedding and metadata.

Chunks are created when chunking is enabled in ExtractionConfig. Each chunk contains the text content, optional embedding vector (if embedding generation is configured), and metadata about its position in the document.

ChunkMetadata¶

Metadata about a chunk's position in the original document.

ChunkingConfig¶

Chunking configuration.

Configures text chunking for document content, including chunk size, overlap, trimming behavior, and optional embeddings.

Use ..the default constructor when constructing to allow for future field additions:

Methods¶

Default()¶

Signature:

func (o *ChunkingConfig) Default() ChunkingConfig

CitationMetadata¶

Citation file metadata (RIS, PubMed, EndNote).

ClassificationLabel¶

A single label + confidence pair.

ContentFilterConfig¶

Cross-extractor content filtering configuration.

Controls whether "furniture" content (headers, footers, page numbers, watermarks, repeating text) is included in or stripped from extraction results. Applies across all extractors (PDF, DOCX, RTF, ODT, HTML, etc.) with format-specific implementation.

When nil on ExtractionConfig, each extractor uses its current default behavior unchanged.

Methods¶

Default()¶

Signature:

func (o *ContentFilterConfig) Default() ContentFilterConfig

ContributorRole¶

JATS contributor with role.

CoreProperties¶

Dublin Core metadata from docProps/core.xml

Contains standard metadata fields defined by the Dublin Core standard and Office-specific extensions.

CsvMetadata¶

CSV/TSV file metadata.

DbfFieldInfo¶

dBASE field information.

DbfMetadata¶

dBASE (DBF) file metadata.

DetectResponse¶

MIME type detection response.

DetectionResult¶

Page-level detection result containing all detections and page metadata.

DiffHunk¶

A single contiguous hunk in a unified diff.

DiffOptions¶

Options controlling how two ExtractionResult values are compared.

Methods¶

Default()¶

Signature:

func (o *DiffOptions) Default() DiffOptions

DjotContent¶

Comprehensive Djot document structure with semantic preservation.

This type captures the full richness of Djot markup, including:

• Block-level structures (headings, lists, blockquotes, code blocks, etc.)
• Inline formatting (emphasis, strong, highlight, subscript, superscript, etc.)
• Attributes (classes, IDs, key-value pairs)
• Links, images, footnotes
• Math expressions (inline and display)
• Tables with full structure

Available when the djot feature is enabled.

DjotImage¶

Image element in Djot.

DjotLink¶

Link element in Djot.

DocumentExtractor¶

Trait for document extractor plugins.

Implement this trait to add support for new document formats or to override built-in extraction behavior with custom logic.

Return Type¶

Extractors return InternalDocument, a flat intermediate representation. The pipeline converts this into the public ExtractionResult via the derivation step.

Priority System¶

When multiple extractors support the same MIME type, the registry selects the extractor with the highest priority value. Use this to:

• Override built-in extractors (priority > 50)
• Provide fallback extractors (priority < 50)
• Implement specialized extractors for specific use cases

Default priority is 50.

Thread Safety¶

Extractors must be thread-safe (Send + Sync) to support concurrent extraction.

Methods¶

ExtractBytes()¶

Extract content from a byte array.

This is the core extraction method that processes in-memory document data.

Returns:

An InternalDocument containing the extracted elements, metadata, and tables. The pipeline will convert this into the public ExtractionResult.

Errors:

• KreuzbergError.Parsing - Document parsing failed
• KreuzbergError.Validation - Invalid document structure
• KreuzbergError.Io - I/O errors (these always bubble up)
• KreuzbergError.MissingDependency - Required dependency not available

Signature:

func (o *DocumentExtractor) ExtractBytes(content []byte, mimeType string, config ExtractionConfig) (InternalDocument, error)

ExtractFile()¶

Extract content from a file.

Default implementation reads the file and calls extract_bytes. Override for custom file handling, streaming, or memory optimizations.

Returns:

An InternalDocument containing the extracted elements, metadata, and tables.

Errors:

Same as extract_bytes, plus file I/O errors.

Signature:

func (o *DocumentExtractor) ExtractFile(path string, mimeType string, config ExtractionConfig) (InternalDocument, error)

SupportedMimeTypes()¶

Get the list of MIME types supported by this extractor.

Can include exact MIME types and prefix patterns:

• Exact: "application/pdf", "text/plain"
• Prefix: "image/*" (matches any image type)

Returns:

A slice of MIME type strings.

Signature:

func (o *DocumentExtractor) SupportedMimeTypes() []string

Priority()¶

Get the priority of this extractor.

Higher priority extractors are preferred when multiple extractors support the same MIME type.

Priority Guidelines¶

• 0-25: Fallback/low-quality extractors
• 26-49: Alternative extractors
• 50: Default priority (built-in extractors)
• 51-75: Premium/enhanced extractors
• 76-100: Specialized/high-priority extractors

Returns:

Priority value (default: 50)

Signature:

func (o *DocumentExtractor) Priority() int32

CanHandle()¶

Optional: Check if this extractor can handle a specific file.

Allows for more sophisticated detection beyond MIME types. Defaults to true (rely on MIME type matching).

Returns:

true if the extractor can handle this file, false otherwise.

Signature:

func (o *DocumentExtractor) CanHandle(path string, mimeType string) bool

DocumentNode¶

A single node in the document tree.

Each node has deterministic id, typed content, optional parent/children for tree structure, and metadata like page number, bounding box, and content layer.

DocumentRelationship¶

A resolved relationship between two nodes in the document tree.

DocumentRevision¶

A single tracked change embedded in a document.

Populated by per-format extractors that understand change-tracking metadata (DOCX w:ins/w:del/w:rPrChange, ODT text:change-*, …). Every extractor defaults to ExtractionResult.revisions = None until a format-specific implementation is added.

DocumentStructure¶

Top-level structured document representation.

A flat array of nodes with index-based parent/child references forming a tree. Root-level nodes have parent: None. Use body_roots() and furniture_roots() to iterate over top-level content by layer.

Validation¶

Call validate() after construction to verify all node indices are in bounds and parent-child relationships are bidirectionally consistent.

Methods¶

FinalizeNodeTypes()¶

Compute and populate the node_types field from the current nodes.

Call this after all nodes have been added to the structure. Internal construction paths (builder, derivation) call this automatically.

Signature:

func (o *DocumentStructure) FinalizeNodeTypes()

IsEmpty()¶

Check if the document structure is empty.

Signature:

func (o *DocumentStructure) IsEmpty() bool

Default()¶

Signature:

func (o *DocumentStructure) Default() DocumentStructure

DocumentSummary¶

Summary of an extracted document.

DocxAppProperties¶

Application properties from docProps/app.xml for DOCX

Contains Word-specific document statistics and metadata.

DocxMetadata¶

Word document metadata.

Extracted from DOCX files using shared Office Open XML metadata extraction. Integrates with office_metadata module for core/app/custom properties.

Element¶

Semantic element extracted from document.

Represents a logical unit of content with semantic classification, unique identifier, and metadata for tracking origin and position.

ElementMetadata¶

Metadata for a semantic element.

EmailAttachment¶

Email attachment representation.

Contains metadata and optionally the content of an email attachment.

EmailConfig¶

Configuration for email extraction.

EmailExtractionResult¶

Email extraction result.

Complete representation of an extracted email message (.eml or .msg) including headers, body content, and attachments.

EmailMetadata¶

Email metadata extracted from .eml and .msg files.

Includes sender/recipient information, message ID, and attachment list.

EmbeddedChanges¶

Changes to embedded archive children between two results.

EmbeddedDiff¶

Diff for a single embedded archive entry that appears in both results.

EmbeddedFile¶

Embedded file descriptor extracted from the PDF name tree.

EmbeddingBackend¶

Trait for in-process embedding backend plugins.

Async to match the convention used by OcrBackend, DocumentExtractor, and PostProcessor. Host-language bridges (PyO3, napi-rs, Rustler, extendr, magnus, ext-php-rs, C FFI, etc.) wrap their synchronous host callables in spawn_blocking or the equivalent to satisfy the async signature.

Thread safety¶

Backends must be Send + Sync + 'static. They are stored in Arc<dyn EmbeddingBackend> and called concurrently from kreuzberg's chunking pipeline. If the backend's underlying model isn't thread-safe, the backend itself must serialize access internally (e.g. via Mutex<Inner>).

Contract¶

• embed(texts) MUST return exactly texts.len() vectors, each of length self.dimensions(). The dispatcher in crate.embeddings.embed_texts validates this before returning to downstream consumers; a non-conforming backend surfaces as a KreuzbergError.Validation, not a panic.

• embed may be called from any thread. Its future must be Send (enforced by async_trait when #[async_trait] is used on non-WASM targets).

• dimensions() is called exactly once at registration, immediately after initialize() succeeds. The returned value is cached by the registry and used for all subsequent shape validation. Lazy-loading implementations can defer model loading into initialize() and report the real dimension afterwards. Later mutations of the backend's reported dimension are not observed by kreuzberg — implementations that need to change dimension must unregister and re-register.

• shutdown() (inherited from Plugin) may be invoked concurrently with an in-flight embed() call. Implementations must tolerate this — e.g. by letting in-flight calls finish using resources held via the Arc<dyn EmbeddingBackend> reference, and only releasing shared state that isn't needed by embed.

Runtime¶

The synchronous embed_texts entry uses tokio.task.block_in_place to await the trait's async embed, which requires a multi-thread tokio runtime. Callers running inside a current_thread runtime (e.g. #[tokio.test] without flavor = "multi_thread", or tokio.runtime.Builder.new_current_thread()) must use embed_texts_async instead, which awaits directly without block_in_place.

Methods¶

Dimensions()¶

Embedding vector dimension. Must be > 0 and must match the length of every vector returned by embed.

Signature:

func (o *EmbeddingBackend) Dimensions() int

Embed()¶

Embed a batch of texts, returning one vector per input in order.

Errors:

Implementations should return Plugin for backend-specific failures. The dispatcher layers its own validation (length, per-vector dimension) on top.

Signature:

func (o *EmbeddingBackend) Embed(texts []string) ([][]float32, error)

EmbeddingConfig¶

Embedding configuration for text chunks.

Configures embedding generation using ONNX models via the vendored embedding engine. Requires the embeddings feature to be enabled.

Methods¶

Default()¶

Signature:

func (o *EmbeddingConfig) Default() EmbeddingConfig

EmbeddingPreset¶

Preset configurations for common RAG use cases.

Each preset combines chunk size, overlap, and embedding model to provide an optimized configuration for specific scenarios.

All string fields are owned String for FFI compatibility — instances are safe to clone and pass across language boundaries.

Entity¶

A single named entity detected in the extracted text.

EpubMetadata¶

EPUB metadata (Dublin Core extensions).

ErrorMetadata¶

Error metadata (for batch operations).

ExcelMetadata¶

Excel/spreadsheet format metadata.

Identifies the document as a spreadsheet source via the FormatMetadata.Excel discriminant. Sheet count and sheet names are stored inside this struct.

ExcelSheet¶

Single Excel worksheet.

Represents one sheet from an Excel workbook with its content converted to Markdown format and dimensional statistics.

ExcelWorkbook¶

Excel workbook representation.

Contains all sheets from an Excel file (.xlsx, .xls, etc.) with extracted content and metadata.

ExtractedImage¶

Extracted image from a document.

Contains raw image data, metadata, and optional nested OCR results. Raw bytes allow cross-language compatibility - users can convert to PIL.Image (Python), Sharp (Node.js), or other formats as needed.

ExtractedUri¶

A URI extracted from a document.

Represents any link, reference, or resource pointer found during extraction. The kind field classifies the URI semantically, while label carries optional human-readable display text.

ExtractionConfig¶

Main extraction configuration.

This struct contains all configuration options for the extraction process. It can be loaded from TOML, YAML, or JSON files, or created programmatically.

Methods¶

Default()¶

Signature:

func (o *ExtractionConfig) Default() ExtractionConfig

NeedsImageProcessing()¶

Check if image processing is needed by examining OCR and image extraction settings.

Returns true if either OCR is enabled or image extraction is configured, indicating that image decompression and processing should occur. Returns false if both are disabled, allowing optimization to skip unnecessary image decompression for text-only extraction workflows.

Optimization Impact¶

For text-only extractions (no OCR, no image extraction), skipping image decompression can improve CPU utilization by 5-10% by avoiding wasteful image I/O and processing when results won't be used.

Signature:

func (o *ExtractionConfig) NeedsImageProcessing() bool

ExtractionDiff¶

The complete diff between two ExtractionResult values.

ExtractionResult¶

General extraction result used by the core extraction API.

This is the main result type returned by all extraction functions.

Methods¶

FromOcr()¶

Convert from an OCR result.

Signature:

func (o *ExtractionResult) FromOcr(ocr OcrExtractionResult) ExtractionResult

FictionBookMetadata¶

FictionBook (FB2) metadata.

FileExtractionConfig¶

Per-file extraction configuration overrides for batch processing.

All fields are Option<T> — nil means "use the batch-level default." This type is used with batch_extract_files and batch_extract_bytes to allow heterogeneous extraction settings within a single batch.

Excluded Fields¶

The following ExtractionConfig fields are batch-level only and cannot be overridden per file:

• max_concurrent_extractions — controls batch parallelism
• use_cache — global caching policy
• acceleration — shared ONNX execution provider
• security_limits — global archive security policy

Footnote¶

Footnote in Djot.

FormattedBlock¶

Block-level element in a Djot document.

Represents structural elements like headings, paragraphs, lists, code blocks, etc.

GridCell¶

Individual grid cell with position and span metadata.

HeaderMetadata¶

Header/heading element metadata.

HeadingContext¶

Heading context for a chunk within a Markdown document.

Contains the heading hierarchy from document root to this chunk's section.

HeadingLevel¶

A single heading in the hierarchy.

HierarchicalBlock¶

A text block with hierarchy level assignment.

Represents a block of text with semantic heading information extracted from font size clustering and hierarchical analysis.

HierarchyConfig¶

Hierarchy extraction configuration for PDF text structure analysis.

Enables extraction of document hierarchy levels (H1-H6) based on font size clustering and semantic analysis. When enabled, hierarchical blocks are included in page content.

Methods¶

Default()¶

Signature:

func (o *HierarchyConfig) Default() HierarchyConfig

HtmlMetadata¶

HTML metadata extracted from HTML documents.

Includes document-level metadata, Open Graph data, Twitter Card metadata, and extracted structural elements (headers, links, images, structured data).

HtmlOutputConfig¶

Configuration for styled HTML output.

When set on html_output alongside output_format = OutputFormat.Html, the pipeline builds a StyledHtmlRenderer instead of the plain comrak-based renderer.

Methods¶

Default()¶

Signature:

func (o *HtmlOutputConfig) Default() HtmlOutputConfig

ImageExtractionConfig¶

Image extraction configuration.

Methods¶

Default()¶

Signature:

func (o *ImageExtractionConfig) Default() ImageExtractionConfig

ImageMetadata¶

Image metadata extracted from image files.

Includes dimensions, format, and EXIF data.

ImageMetadataType¶

Image element metadata.

ImagePreprocessingConfig¶

Image preprocessing configuration for OCR.

These settings control how images are preprocessed before OCR to improve text recognition quality. Different preprocessing strategies work better for different document types.

Methods¶

Default()¶

Signature:

func (o *ImagePreprocessingConfig) Default() ImagePreprocessingConfig

ImagePreprocessingMetadata¶

Image preprocessing metadata.

Tracks the transformations applied to an image during OCR preprocessing, including DPI normalization, resizing, and resampling.

InlineElement¶

Inline element within a block.

Represents text with formatting, links, images, etc.

JatsMetadata¶

JATS (Journal Article Tag Suite) metadata.

Keyword¶

Extracted keyword with metadata.

KeywordConfig¶

Keyword extraction configuration.