Local first.

Developer laptops are powerful. Most of them are way more powerful than most of the servers running out there. Having things locally is a great way to have control and avoid surprises.

In the original Tinybird design, we focused on the cloud and only designed a small part of the platform to work locally. We've received a lot of feedback about this, based on that feedback, we decided to shift development with Tinybird to be local first.

So, we introduce Tinybird Local, a Docker container with a full image of Tinybird that you can run on your laptop.

Now, you have everything you need to develop, test, and ship your data applications locally. You could even ship to production on your laptop (we know you've done it before don't lie).

We decided to take an unusual path: instead of having a different version of the platform to run locally, we created a container that allows you to have a full local instance of Tinybird. It is almost the exact same thing we have been running on the cloud for over 5 years.

In fact, you can even run Tinybird Local in production (more on that later this week).

The TL;DR? You can now deploy your data applications locally or to the cloud in the exact same way, which is the way things should be.

How do I use Tinybird Local?

We manage the container for you, so you don't need to worry about it. We check for updates when you start the container, and we take care of everything.

You can check if it's running with docker ps, or you can start the container manually with:

Read the full Tinybird Local documentation here.

More on why we created Tinybird Local 

Allow me to provide some more details on why we went local:

1. Development and testing: Developers need a way to test their data pipelines locally before deploying to production. Tinybird Local makes that much easier than before.
2. CI/CD integration: This is actually where the idea was spawned. Canva wanted to test their data pipelines in their CI without using any cloud service. You can deploy Tinybird Local in your CI environments to do this.
3. Learning and experimentation: New users want a risk-free environment to explore Tinybird's capabilities. Nobody wants that feeling of "I could accidentally drop a 5Tb table with this change." With Tinybird Local, that feeling doesn't disappear, but you still have much more confidence when experimenting.
4. Offline development: Sometimes you're on a train, plane, or beach. There are still scenarios where you need to work without an internet connection. While these scenarios are much less common than they were 10 years ago, it's still very useful to have a fast local container in case you have a slow or spotty internet connection.

What's under the hood?

Tinybird Local runs a streamlined version of our platform in a Docker container. It includes:

• The core Tinybird data processing engine
• API endpoints for data ingestion and querying
• CLI compatibility for workflow management

However, it doesn't include some features of the full cloud version:

• Tinybird Local doesn't include a local version of the Tinybird UI, but you can use the UI in cloud.tinybird.co to visualize your local instance with the tb open command.
• Scheduled operations like sinks and copies

The container is designed to be ephemeral - you can destroy and recreate it as needed, making it perfect for testing and development workflows.

Tinybird Local consists of several key components:

1. ClickHouse Engine: A streamlined version of the same analytical database that powers our cloud offering. It has some key benefits over the open source version, which we'll talk about it in future posts.
2. API Layer: A fully compatible implementation of our REST APIs for data ingestion and querying.
3. Ingestion APIs: You can handle ingestion (streaming and batch) with local versions of our /events and /datasources APIs.
4. Batch operations API: Batch operations are hard, but this API makes it easy.
5. Authentication System: A token-based authentication system, including JWTs to enable quick integration with other services.

Technical implementation details

We made several key architectural decisions to optimize Tinybird Local for developer experience:

Simplified architecture

• CHKeeper instead of ZooKeeper: We replaced ZooKeeper with CHKeeper to reduce the number of services and container size. This simplified dependency management while maintaining the core functionality needed.
• Supervisor for process management: We use Supervisor to manage all internal processes within the container, making it easier to start, monitor, and restart services as needed.
• Nginx as Reverse Proxy: We added Nginx to handle routing and provide a unified entry point for all API requests, simplifying the networking model.

Performance optimizations

• CPU usage controls: We've disabled several background processes and optimized configurations to prevent the container from consuming excessive CPU on your development machine.
• Reduced memory footprint: We stripped unnecessary ClickHouse components, reducing the container size from 5GB to a much more manageable footprint.
• Faster startup times: By removing non-essential services, we've significantly reduced the time it takes for the container to become fully operational.

Developer experience enhancements

• /tokens Endpoint: We created a dedicated endpoint for token management, ensuring that developers always have a valid token to start working with (rather than needing to go fetch one from the UI), eliminating a common friction point.
• Auto-configuration: The container automatically configures itself on startup, so you don't need to perform any manual setup steps.
• Compatibility Layer: We've implemented compatibility shims for cloud-specific features, allowing most code to work unchanged between local and cloud environments.

Technical limitations

Understanding these technical details helps explain some limitations:

• Single-node execution means some distributed queries behave differently
• Memory constraints on local machines limit the size of datasets you can process
• Some cloud-specific features rely on services not available locally

This architecture ensures that while Tinybird Local runs on your laptop, it maintains compatibility with our cloud platform where it matters most.

The future of Tinybird Local

We're just getting started with Tinybird Local, and we have an exciting roadmap ahead. Here's a glimpse of what we're working on:

• Better storage options, like cloud storage. We run a fork of ClickHouse optimized to work on cloud storage (it has been running for +1 year in our cloud).
• Backups
• Monitoring
• Settings configuration. There are many things you want to control depending on the use case.
• More deployment options.

Community-driven development

Our roadmap is heavily influenced by developer feedback. We're actively collecting input on what would make Tinybird Local even more valuable for your workflows.

Have a feature request or idea? We'd love to hear from you! Share your thoughts in our Slack community.

FAQ

Can I deploy Tinybird Local to production?

Yes, check out self-managed Tinybird Cloud here.

What are the resource requirements for running Tinybird Local?

Tinybird Local is designed to be lightweight, but we recommend at least 4GB of RAM and 2 CPU cores for a smooth experience.

Is my data persistent in Tinybird Local?

Data in Tinybird Local persists as long as the container exists. If you remove the container, all data will be lost. For development purposes, this is usually desirable as it gives you a clean slate.

You can however set persistent volumes for your data. Check out the documentation for persistent volumes here.

Can I connect Tinybird Local to external data sources?

Yes, you can connect to external data sources that are accessible from your local machine. Currently, Kafka and S3 are supported.

How does Tinybird Local differ from the cloud version?

Tinybird Local and Tinybird Cloud are very similar, but Tinybird Cloud, by nature, removes single node limitations and scalability issues associated with running Tinybird on a single, local machine.

Limitations and considerations

• Performance will depend on your local hardware
• Network connectivity may be required for certain operations (such as authentication or downloading the latest local image)
• The container isn't updated automatically. You'll be prompted to download new versions when you start the container. To update a running container, you can use tb local restart.

More resources

• Tinybird Local documentation
• Tinybird Local tutorial video
• Tinybird CLI reference
• Ask us anything about Tinybird local (Google Cal Link) (this Friday)
• GitHub repository (if applicable)