Changelog #19: Adding new columns in high-frequency ingestion and more

As the days get longer, so does our list of product improvements! Here’s what we’ve been up to over the past few weeks.

Add new columns during high-frequency ingestion

With high-frequency ingestion, you can ingest NDJSON data at a rate of hundreds of requests per second. But, if new columns appear in the data stream, there will be a mismatch between the schema of the Data Source and the data stream.

If this happens, you’ll be prompted to add new columns in your product dashboard. You can choose which columns you wish to keep, and also change the data types from the suggested types if required. Check out our guide to learn more about high-frequency ingestion.

Pass parameters across pipes

It’s efficient to reuse pipes, and if these pipes use parameters, it makes sense for those parameters to be passed from other pipes. We’ve made some changes so passing parameters across pipes now works out of the box. You no longer need to copy and paste complex nodes from one pipe to another or use INCLUDE statements.

In the example below, the endpoint of the second pipe is queried with the API call. The parameters for the time period and the wiki are specified in the call. The filtering in the first pipe uses the wiki parameter.

curl -X GET “https://api.tinybird.co/v0/pipes/pipe_2.json?wiki=wikidatawiki&date_start=2022-03-01+10%3A30%3A00&date_end=2022-03-01+11%3A00%3A00&token=p…”

New and improved materialized view icons

Sometimes it’s hard to know the types of the Data Sources and pipes you’re building with in the UI. So we made it easier to tell what’s what at a glance. The updated icons show which pipes expose a materialized view and which Data Sources are materialized views. You’ll find these icons in the sidebar, the data flow, the Data Sources modal, and the Data Source log.

CLI updates!

Materialized views - Materialized views can be tricky to create. We’re working to make it easier to create error-free materialized views in the CLI. The command ‘tb materialize path/to/pipe.pipe node_name’ best guesses the target Data Source by analyzing the node_name SQL query to automatically generate:

• the target .datasource file needed to push a new materialized view
• a version of the .pipe that automatically completes the aggregate functions with -State combinators where appropriate, so you don’t have to remember to include them.

You can find more details about materialized views in the CLI, here in our docs.

Perform regression tests - Improving the performance of your data project shouldn’t mean changing the results. We’ve added a new CLI command so you can perform a regression test and know if there are any differences. Run ‘tb pipe regression-test –help’ to see all the options.

Check out the latest command-line updates in the changelog.

ClickHouse improvements

We’ve helped speed up the performance of the ANY aggregation function by 15-30% by ensuring it acts over batches and stops processing data as soon as an element has been saved.

Community Slack

If you want to be part of the conversation, you can join our community in Slack. We’d love to hear what you’re building with Tinybird!