The Data Base

Educational content about databases, data warehouses, or other data infrastructure. All posts about ClickHouse should be included here. All posts about Postgres, MySQL, MongoDB, etc. should go here. If it has database in the title, it goes here.

The Data Base

Lessons learned from 5 years operating huge ClickHouse® clusters: Part II

This is the second part of the series. Here's more of what I've learned from operating petabyte-scale ClickHouse clusters for the last 5+ years.

The Data Base

I've operated petabyte-scale ClickHouse® clusters for 5 years

I've been operating large ClickHouse clusters for years. Here's what I've learned about architecture, storage, upgrades, config, testing, costs, and ingestion.

The Data Base

We added the Backup Database engine to ClickHouse

We added a new ClickHouse engine: Backup. Here's why we did it, how it's implemented, and example usage.

The Data Base

Outgrowing Postgres: How to optimize and integrate an OLTP + OLAP stack

Navigate the complexities of OLTP and OLAP integration by choosing simple, scalable data movement patterns that reduce infrastructure overhead and keep your focus on building great products for users.

The Data Base

Outgrowing Postgres: How to evaluate the right OLAP solution for analytics

Moving analytical workloads off Postgres? Learn how to evaluate real-time OLAP solutions based on what actually matters: performance, SQL compatibility, and developer experience.

The Data Base

Outgrowing Postgres: When to move OLAP workloads off Postgres

Learn when to move analytics off Postgres by watching for technical and team health warning signs before crisis hits.

The Data Base

Outgrowing Postgres: How to run OLAP workloads on Postgres

A deep dive into running analytics on Postgres, from basic optimizations to advanced techniques and knowing when to quit.

The Data Base

Outgrowing Postgres: Handling increased user concurrency

When your application grows, so too do your database connections. Learn how to handle increased user concurrency on Postgres.

The Data Base

Outgrowing Postgres: Handling growing data volumes

Managing terabyte-scale data in Postgres? From basic maintenance to advanced techniques like partitioning and materialized views, learn how to scale your database effectively. Get practical advice on optimizing performance and knowing when it's time to explore other options.

The Data Base

Outgrowing Postgres: How to identify scale problems

Discover early warning signs that you’ve outgrown PostgreSQL and learn how to keep performance high. This introductory article offers diagnostic techniques and proactive strategies to help you scale and plan the future of your analytics without losing momentum.

The Data Base

Query DynamoDB tables with SQL

Want to aggregate, filter, or join DynamoDB tables with SQL? Here's how to do it, and why you should (and shouldn't) query DynamoDB tables with SQL.

The Data Base

Top Use Cases for DynamoDB in 2024

DynamoDB… it's fast, scalable, and flexible. What's not to love? Here are the top use cases for DynamoDB in 2024 (and a few areas where it won't work).

The Data Base

Want a managed ClickHouse®️? Here are some options

A managed ClickHouse service can speed up your development and reduce the infrastructure you need to set up and maintain. But what managed ClickHouse service should you choose? Here are your options.

The Data Base Featured

Tinybird vs. ClickHouse®️: What's the difference?

Tinybird is a real-time data platform for user-facing analytics, built using ClickHouse. Here are the differences between Tinybird and other ClickHouse solutions, including self-hosted and managed.

The Data Base

Best practices for timestamps and time zones in databases

Confused about how to handle dates, times, and DateTimes in your database? We’ve got you covered, now() and always.

The Data Base

ClickHouse®️ JOINs... 100x faster

We recently introduced two pull requests to ClickHouse that significantly improve JOIN performance in common scenarios.

The Data Base

What is the best database for real-time analytics?

These are the 3 best databases for real-time analytics, and how to avoid challenges when deploying them.

The Data Base

What are columnar databases? Here are 35 examples.

New to columnar databases? Read this article to learn what a columnar database is, when to use it, and popular examples of columnar databases.

The Data Base

A practical guide to real-time CDC with MySQL

A step-by-step guide to setting up Change Data Capture (CDC) with MySQL, Confluent Cloud, and Tinybird.

The Data Base

A practical guide to real-time CDC with Postgres

A step-by-step guide to setting up Change Data Capture (CDC) with PostgreSQL, Confluent Cloud, and Tinybird.

The Data Base

Using Bloom filter indexes for real-time text search in ClickHouse®️

A customer of ours had text-based log data and they wanted to be able to search over the text (quickly). However, in ClickHouse, text search without any special measures involves a full scan, period. And we know that full scans are not efficient.

The Data Base

5 Snowflake struggles that every Data Engineer deals with

Snowflake is the world’s leading cloud data warehouse, but it is almost always slow and costly for application development. Tinybird makes it easy to quickly and cost-effectively build applications on top of your Snowflake data. Tinybird and Snowflake are better together.

The Data Base

Real-time Databases: What developers need to know

Explore the key factors in choosing a real-time database and compare MongoDB, PostgreSQL, Tinybird, ClickHouse, Snowflake, Pinot, and Druid to determine the best fit for your application's performance, scalability, and analytics needs.

The Data Base

Understanding the Data Warehouse

What gave rise to the Data Warehouse? What do they enable? I answer these questions in this first of a two-blog series.

The Data Base

When you should use columnar databases and not Postgres, MySQL, or MongoDB

Row-oriented, OLTP databases aren't ideal application DBs when you know you'll need to run analytics on lots of data. Choose a column-oriented OLAP instead.