Development Workflow

Everything you need to build, test, and debug Kreuzberg locally. This guide assumes you’ve already followed the Contributing Guide to fork and clone the repo.

---

The Task Runner

Kreuzberg uses Task for all build and test workflows. One command to bootstrap everything:

Terminal

task setup

That installs all toolchains and dependencies. Safe to re-run anytime — it’s idempotent.

The Pattern

Tasks follow <language>:<action>. Once you learn this pattern, the command for any task is predictable:

Terminal

task rust:build           # Build the Rust core
task rust:build:dev       # Debug build (faster compile, no optimizations)
task rust:build:release   # Release build (slow compile, fast binary)
task rust:test            # Run Rust tests
task rust:test:ci         # Same tests, with CI-level diagnostics

task python:build         # Build Python bindings via maturin
task python:test          # Run Python test suite
task node:build           # Build Node.js bindings via napi
task node:test            # Jest tests

The same pattern works for every language: go:build, java:test, ruby:build, csharp:test, and so on.

Bulk Operations

Terminal

task build:all            # Build every binding
task test:all             # Test every binding (sequential)
task test:all:parallel    # Test every binding (parallel — faster, noisier output)
task check                # Lint + format check across the whole repo

---

Testing Locally

Rust

The core lives in crates/kreuzberg/. Most changes start here.

Terminal

task rust:test

cargo test -p kreuzberg test_pdf_extraction -- --nocapture

RUST_LOG=debug cargo test -p kreuzberg test_name -- --nocapture

Python

Python bindings are in packages/python/. Build first, then test:

Terminal

task python:build:dev
task python:test

cd packages/python
uv run pytest tests/ -k "test_extract" -v

The RUST_LOG env var works here too — the Rust core logs through Python’s stderr:

Terminal

RUST_LOG=debug uv run pytest tests/ -v

Node.js

TypeScript bindings are in packages/typescript/:

Terminal

task node:build:dev
task node:test

cd packages/typescript
pnpm test -- --testPathPattern="extract"

Everything Else

Same pattern. Build, then test:

Terminal

task go:build && task go:test
task java:build && task java:test
task csharp:build && task csharp:test
task ruby:build && task ruby:test
task php:build && task php:test
task elixir:build && task elixir:test
task r:build && task r:test
task c:build && task c:test
task wasm:build && task wasm:test

---

E2E Test Suites

End-to-end tests guarantee that every language binding produces identical results for the same document. They live in e2e/ as shared fixtures — test inputs paired with expected outputs.

Run E2E Tests

Language	Directory	Run with
Python	e2e/python/	task python:e2e:test
TypeScript / Node.js	e2e/typescript/	task node:e2e:test
Rust	e2e/rust/	task rust:e2e:test
Go	e2e/go/	task go:e2e:test
Java	e2e/java/	task java:e2e:test
.NET	e2e/csharp/	task csharp:e2e:test
Ruby	e2e/ruby/	task ruby:e2e:test
PHP	e2e/php/	task php:e2e:test
R	e2e/r/	task r:e2e:test

Regenerate E2E Tests

When you add a feature that changes extraction behavior, regenerate the affected E2E suites:

Terminal

task python:e2e:generate
task node:e2e:generate
task <lang>:e2e:generate

To regenerate and test all suites at once:

Terminal

task e2e:generate:all
task e2e:test:all

---

Linting and Pre-commit

Terminal

task check              # Full lint + format check (same as CI validate stage)

Language-specific:

Terminal

task rust:lint          # clippy + rustfmt
task python:lint        # ruff + pyrefly
task node:lint          # poly (oxc) lint + format

The repo uses poly for formatting and lint rules — run poly fmt --check . and poly lint . (or task lint) before committing. CI enforces the same checks plus conventional commit messages; if CI fails, the output tells you exactly what to fix.

---

Working with Documentation

Building Locally

Terminal

task docs:build
task docs:serve

How Snippets Work

Code examples in the docs aren’t inline — they’re pulled from docs/snippets/ via a snippet-include directive. This keeps examples testable and reusable across pages.

docs/snippets/
├── python/           # Python examples
│   ├── api/          #   extract_file, batch_extract, etc.
│   ├── config/       #   ExtractionConfig, OcrConfig, etc.
│   ├── ocr/          #   OCR backends
│   ├── plugins/      #   Plugin implementations
│   ├── mcp/          #   MCP server and client
│   └── utils/        #   Embeddings, chunking, errors
├── rust/             # Rust examples (same layout)
├── typescript/       # TypeScript examples
├── go/, java/, csharp/, ruby/, r/
├── docker/           # Docker commands
├── api_server/       # Server startup examples
└── cli/              # CLI usage

When you change a user-facing API, update the matching snippet. When you add a new feature, create a snippet and include it from the relevant doc page.

---

Debugging

Rust Panics

Terminal

RUST_BACKTRACE=1 cargo test -p kreuzberg test_name
RUST_BACKTRACE=full cargo test -p kreuzberg test_name

Python FFI Problems

When something goes wrong in the Rust core during a Python call, the error introspection API gives you the details:

debug_ffi.py

from kreuzberg import get_last_error_code, get_error_details, get_last_panic_context

details = get_error_details()
print(f"Error: {details['message']}")
print(f"Code: {details['error_code']}")

context = get_last_panic_context()
if context:
    print(f"Panic context: {context}")

Verbose Logging

Crank up the log level to see what the Rust core is doing:

Terminal

RUST_LOG=debug task python:test
RUST_LOG=trace task rust:test

---

CI/CD

CI runs on every push and PR to main via .github/workflows/ci.yaml. The pipeline has four stages:

1. Validate — conventional commits, formatting, clippy
2. Build — FFI libraries, Python wheels, Node packages, all bindings
3. Test — per-language test suites on Linux, macOS, and Windows
4. Integration — Docker build, Docker smoke tests, CLI tests

Smart Change Detection

CI doesn’t rebuild everything on every PR. A changes job detects which paths were touched and only runs the relevant build/test jobs. Edit a Python file? Only Python builds and tests run. Touch the Rust core? Everything downstream rebuilds.

Running CI Checks Locally

Before pushing, you can run the same checks CI runs:

Terminal

task check              # Matches the validate stage
task rust:test:ci       # Rust tests with CI diagnostics
task python:test:ci     # Python tests with CI diagnostics
task test:all:ci        # Everything

Other Workflows

Workflow	When it runs	What it does
ci.yaml	Every push/PR to main	The main pipeline
docs.yaml	Changes to docs-site/**	Builds and deploys documentation
publish.yaml	Release events	Publishes packages to registries
publish-docker.yaml	Tags and releases	Builds and pushes Docker images

---

Performance

Kreuzberg’s core is written in Rust, which enables zero-copy memory handling, SIMD acceleration, and true multi-core parallelism — all at compile time with no garbage collection.

Why Rust Matters

• Native compilation: LLVM optimizes code ahead of time (inlining, vectorization, dead code elimination)
• Zero-copy strings: Slicing uses borrowed references, not heap allocations
• SIMD acceleration: Whitespace detection and character classification run 15-37x faster than scalar operations
• No GIL: True multi-core parallelism across all CPU cores
• Deterministic memory: Drop semantics free memory instantly, no GC pauses

Key Optimizations

• Batch processing: 6-10x faster than sequential extraction through work-stealing scheduler
• Caching: 85%+ hit rates for repeated files (SQLite-backed, automatic invalidation)
• Streaming: Large files processed in 4KB chunks, constant memory regardless of file size
• Lazy initialization: Expensive subsystems (Tokio, plugins) initialized on first use only

---