Marketplace
Reference library
Official documentation, specifications, and books worth pointing at — scored on quality, not popularity.
A DuckDB Database with No Data in It
A DuckDB database file does not have to hold any data.
Details
Agent Skills (open format)
The Agent Skills format specification — a SKILL.md file with front-matter (name, description, version), plus references/, scripts/, and examples/ directories that an agent loads on demand. The format this platform's Skill object is modelled on — portable across compatible agents. Covers the SKILL.md schema, the directory layout convention, and the agent loading protocol.
Details
Apache Airflow Documentation
Airflow's documentation — DAG authoring, scheduling semantics and timetables, backfills and catchup, task state and retries, dynamic task mapping, and best practices. The scheduling and best-practices pages answer the idempotency and backfill questions our orchestration skills only sketch; read them before designing any recurring pipeline.
Details
Apache Arrow Documentation
Official Arrow docs: the columnar in-memory format, its language bindings, IPC and Flight protocols, and zero-copy interchange between systems. Use it for exact memory-layout and type-system semantics behind every "columnar" claim in the registry.
Details
Apache Iceberg Documentation
Iceberg's specification and documentation — the table format, snapshots and manifests, hidden partitioning, schema and partition evolution, and maintenance procedures like compaction and snapshot expiry. The table format is a specification, so the spec is the ground truth when engines disagree; essential reading for multi-engine shops where Spark, Trino, and warehouses share tables.
Details
Apache Kafka Documentation
Kafka's official documentation — architecture and design, delivery semantics, the protocol, consumer groups and offsets, operations, and security. The design section explains why ordering and partitions behave the way they do; the operations section companions our lag-triage skill and consumer-lag runbook when a group falls behind.
Details
Apache Parquet Documentation & Spec
Apache Parquet documentation and specification — columnar encoding, row groups, statistics, predicate pushdown, and how to structure files for efficient reads. Understanding statistics and row-group sizing is the practical step that turns "use Parquet" into "and lay out the file like this"; pairs with small-files and lambda-vs-kappa guides.
Details
Apache Parquet Format Documentation
The Parquet file format documentation: page structure, encodings, compression codecs, statistics and the predicate-pushdown behavior that columnar readers rely on. The ground truth for why columnar files scan less data and how row-group pruning works.
Details
Apache Spark Documentation
The authoritative reference for Spark SQL, the DataFrame API, tuning and the configuration catalogue, versioned per Spark release. Use it for exact semantics and config keys — when a job skews or slows, check the config reference and the tuning guide before changing code, alongside our skew-diagnosis and cost skills.
Details
AWS Architecture Center
AWS's official collection of architecture diagrams, reference architectures and best-practice guidance (Well-Architected pillars, serverless data lake patterns). Concrete vendor material for grounding sizing and tradeoff arguments in real deployed patterns.
Details
AWS Glue Documentation
complete reference for Glue jobs, crawlers, the Data Catalog, job bookmarks, DPU configuration, and Ray integration. The job bookmarks and catalog behavior sections are the source of the most common Glue surprises; the Spark UI integration pages explain how to debug slow jobs. Essential for our Glue review skills and CDC pipeline patterns.
Details
Azure Cloud Design Patterns
Microsoft's catalogue of cloud design patterns: retry, circuit breaker, compensating transaction, CQRS, event sourcing, bulkhead and more — each with problem/solution and considerations. Vendor-neutral in practice, and directly applicable to pipeline component design.
Details
BigQuery Documentation
BigQuery's official documentation — comprehensive coverage of partitioning, clustering, slot management, and the query optimizer's behavior. Covers BigQuery ML, BI Engine, materialized views, and authorized views. The partitioning/clustering pages explain the exact levers for cost control; the query optimizer section explains slot utilization and query planning. Essential for our Athena/BigQuery cost comparison guide and BigQuery cost-optimization patterns.
Details
Cost-effective ETL with DuckDB and Amazon S3 Tables on AWS Glue
Learn how to pair DuckDB with AWS Glue 6.0 to run SQL-centric ETL on a single worker, reading Parquet from Amazon S3 and writing Apache Iceberg tables to Amazon S3 Tables.
Details
Daily Life as a Jenkins Administrator
Jenkins Kubernetes setup As a team, we introduced Jenkins Kubernetes for every user a few months ago.
Details
Daily Life as a Jenkins Administrator
Agent ecosystem overview The second blog post focused on one specific part of your Kubernetes environment.
Details
Data Mesh (O'Reilly)
Organizational principles for analytical data at scale — domain ownership, data as a product, self-serve data platform, and federated computational governance. Zhamak Dehghani's Data Mesh provides the organizational counterpoint to a central platform team; the vocabulary and principles are essential for governance discussions and platform strategy. The four principles (domain ownership, data as product, self-serve platform, federated governance) map directly to our governance and platform patterns.
Details
Databricks Documentation
comprehensive coverage of Delta Live Tables, Unity Catalog, jobs and clusters, liquid clustering, Photon engine, and Unity Catalog governance. The Liquid Clustering and Unity Catalog pages are platform-specific and only documented here; the Delta Live Tables section is the definitive reference for building reliable pipelines. Essential for our lakehouse skills and Databricks-specific patterns.
Details
DataHub Documentation
DataHub's documentation — the open metadata model for ownership and lineage, ingestion framework, search, and governance integration. Explains how to represent ownership and lineage consistently across tools; the key vocabulary resource for any catalog-related pipeline design.
Details
dbt Best Practices Guide
dbt Labs' official best practices guide — project structure, staging/marts layering, naming conventions, testing conventions (unique, not_null, relationships, accepted_values), model contracts, and incremental model patterns. A widely-adopted convention set; agreeing on it early avoids enormous friction later. The testing conventions section directly backs our dbt test writer skill; the materialization guide supports our warehouse testing patterns.
Details
dbt Documentation
dbt's documentation — the analytics-engineering workflow, project structure with models and sources, materialisations and incremental-model patterns, tests (unique, not-null, relationships, accepted values) and the docs-as-data approach. The materialisation and incremental-model pages are the reference behind our dbt review and testing skills; pair with our warehouse-testing guide for the practical path.
Details
Debezium Documentation
Debezium's documentation — source and sink connectors, the change-event envelope format, initial snapshots, schema history handling, and the operational pitfalls of running connectors in production. The connector matrix and the events section are what you need to design a correct CDC merge; pair with our Postgres CDC lakehouse project for the full path.
Details
Delta Lake Documentation
Delta Lake's official docs — ACID transactions on object storage, time travel, OPTIMIZE and Z-ORDER clustering, Delta Live Tables, and Unity Catalog integration. The definitive source for Delta features and the maintenance operations our lakehouse skills reference; compare with our Delta vs Iceberg decision guide before committing to a format.
Details
Designing Data-Intensive Applications (O'Reilly)
The canonical book on the principles behind data systems: reliability, scalability, replication, partitioning, consistency, and batch/stream processing. Martin Kleppmann's DDIA is the single highest-value read for system-design interviews and for understanding why data tools are shaped as they are. Every chapter maps directly to patterns in this registry — replication to CDC, partitioning to lakehouse, consistency to transaction semantics. The definitive textbook for data engineers.
Details
DuckDB and Hugging Face: Querying Datasets Directly
Hugging Face hosts hundreds of thousands of datasets, and DuckDB can read them directly, exactly where they are and without downloading anything, over the DuckDB `hf://` protocol.
Details
DuckDB Documentation
DuckDB's documentation — the embedded analytics SQL engine, Parquet and CSV readers, in-process query execution, and extensions (ICU, JSON, Parquet, Excel). The fastest way to make a pipeline testable locally; covers everything our SQL review and testing skills reference, plus the DuckDB docs themselves.
Details
Fundamentals of Data Engineering (O'Reilly)
A lifecycle view of data engineering — generation, storage, ingestion, transformation, and serving — with the "undercurrents" of security, governance, and cost. Joe Reis and Matt Housley provide a modern, practitioner-oriented map of the field. The best overview for someone moving into data engineering; pairs perfectly with this platform's roadmap. Covers the full lifecycle from ingestion through serving, with practical patterns for each stage.
Details
Google SRE Books
Google's Site Reliability Engineering books, free online: SLIs/SLOs/error budgets, incident management, postmortem culture and capacity planning. The vocabulary source for reliability engineering that data platform teams borrow from when they define data SLAs and run incident drills.
Details
Grafana Documentation
Grafana's official documentation: dashboards, panel queries, alerting (including the unified alerting model), and provisioning. The standard surface where pipeline freshness, volume and job-health dashboards live.
Details
Great Expectations Documentation
expectations (expectations, suites, checkpoints) as a YAML DSL for data quality, plus the data-doc approach that connects quality rules to documentation. The expectation catalogue is a practical vocabulary for expressing rules with precision; pairs with our Soda Core reference and data-quality starter guide.
Details
GSoC 2026 Final Blog: How It Was to Be a Contributor at Jenkins
Hello everyone!
Details
Importing Data using Java Table Functions
When the analytics screen running on our main operational database got too slow, we moved a year of data into DuckDB on the same server.
Details
Investigating Build Regressions From the Last Known-Good Build
The following scenario is a case study.
Details
Jenkins and AI Agents: The Trusted Engine for Faster Software Delivery
The 2026 Reality Check By 2026, software development has crossed a threshold.
Details
Jev and DuckDB: Plain-English Conditions in SQL
TypeSafe AI released Jev, a model that returns typed answers instead of text, on September 15, 2026.
Details
kitfunso/hippo-memory
Stop re-teaching your agent.
Details
Materialize once, query anywhere: Introducing Iceberg materialized views in Amazon Redshift
Amazon Redshift now supports Iceberg materialized views.
Details
MCP Registry (official)
The official MCP Registry — community-driven registry of MCP servers with a metadata schema for discovery. The canonical place to discover MCP servers — treat entries as *discovery*, never as a trust signal. The metadata schema defines server capabilities, transport, and authentication; essential for building MCP-aware tooling.
Details
Model Context Protocol Specification
definitive reference for transports (stdio, SSE, HTTP), tool definitions, resource templates, prompts, and the security model. The security section is required reading before connecting any MCP server to real data; the transport section explains stdio vs HTTP tradeoffs. The canonical reference for all MCP server implementations on this platform.
Details
Monitoring MWAA-orchestrated ETL pipelines with Amazon OpenSearch Service
Troubleshooting a multi-service, MWAA-orchestrated ETL pipeline often means hunting across scattered Amazon CloudWatch log groups.
Details
oaslananka/kicad-mcp-pro
MCP server for KiCad, connecting AI agents to schematic and PCB workflows, ERC/DRC validation, DFM checks, BOM analysis, and manufacturing preparation—with controlled edits and human-gated release wor
Details
OpenLineage Documentation
the open specification for lineage events, plus integrations for Airflow, Spark and dbt. Not a tool or a service; it is a standard for emitting lineage events, which is what makes cross-tool lineage possible. Read it before wiring any line-tracking pipeline.
Details
OpenTelemetry Documentation
OpenTelemetry's official documentation: traces, metrics and logs through a vendor-neutral API/SDK, the collector, and semantic conventions. Increasingly the default instrumentation layer for distributed pipelines and services.
Details
Optimize consumer rebalancing on Amazon MSK with next generation protocol
The KIP-848 consumer protocol in Apache Kafka 4.0 redesigns consumer group rebalancing to eliminate stop-the-world pauses.
Details
PostgreSQL Documentation — Tutorial & Reference
PostgreSQL's official documentation: the tutorial, SQL reference, indexes, transactions and MVCC chapters. The semantics of transaction isolation and index behavior here are the ground truth for correctness arguments in pipeline source systems.
Details
Prometheus Documentation
Official Prometheus docs: the pull-based metrics model, PromQL, exporters, service discovery and alerting rules. Data platforms lean on it for infrastructure- and job-level metrics that sit underneath data-quality signals like freshness and volume.
Details
Python Official Documentation
The official Python 3 documentation: language reference, standard library (itertools, collections, dataclasses, json, csv), and the data-model chapter that explains iteration, mutability and hashing semantics every data script depends on.
Details
Query across accounts and table formats with multi-catalog in Amazon EMR
Amazon EMR 8.1.0 adds multi-catalog support through the RedirectingSessionCatalog, so you can query Iceberg, Delta Lake, Hudi, and Hive tables through a single catalog and join data across AWS accounts without copying it.
Details
Reading CI Failures Across Pipelines, Not One Build at a Time
Jenkins is good at explaining a single build.
Details
Snowflake Documentation
Snowflake's official documentation — comprehensive coverage of micro-partitions, clustering keys, virtual warehouses, streams and tasks, time travel, and zero-copy cloning. The performance and cost sections are the definitive reference for clustering-key design, warehouse sizing, and query optimization. The Snowflake Cortex and ML functions sections document built-in ML capabilities. The security chapter covers RBAC, network policies, and Tri-Secret Secure. Essential reading for anyone operating Snowflake at scale; pairs with our cost-engineering and warehouse-clustering skills.
Details
Soda Core
open-source checks-as-code for data quality with a YAML DSL for column and row-level checks, metrics, and thresholds. The SQL-metric approach is easier to reason about than Great Expectations' expectation suites; the programmatic API enables CI/CD integration. Lightweight alternative to Great Expectations; pairs with our data-quality starter guide and Great Expectations reference.
Details
SQLBolt — Learn SQL with Simple Interactive Exercises
Free interactive SQL lessons: SELECT basics, filtering, joins, NULL handling and aggregate clauses, each runnable in the browser. A zero-setup first step before touching a real warehouse or local DuckDB.
Details
The System Design Primer
The widely-used open-source study guide for system design: scaling vocabulary (load balancers, caching, sharding), back-of-envelope estimation, and worked design exercises. Data platform architectures are a special case of the same throughput/consistency tradeoffs it teaches.
Details
The Twelve-Factor App
The classic twelve principles for building services that run portably: config in the environment, backing services as attached resources, logs as event streams, disposability and dev/prod parity. Data pipelines inherit these ideas whenever they run as jobs on shared infrastructure.