Analyzing our own Nginx logs with ClickHouse on Tinybird

In a recent post we walked through the steps required to use Tinybird to analyze Nginx logs. In this one we’ll share how we are using our own product internally to analyze our own Nginx traffic in real-time and automatically, and how you can do it too.

To follow along, you’ll need 3 things:

• A Tinybird account. Sign in if you don’t have one yet.
• The code from this GitHub repo: it contains a Tinybird project Data source configurations and Pipes that will work out of the box.
• tbtail, a tool to stream your logs to Tinybird, installed in the machines where your Nginx logs are generated.

With that, you can quickly get a sense of what’s going on with all your traffic in real-time, as well as examples for you to start slicing and dicing your logs in however other ways you want.

We’re still in private beta, so after you sign in with Google, Github or Microsoft authentication, your account won’t be active. Email us at hi@tinybird.co and we’ll activate it for you.

Run these commands to clone the data project locally, replicate it on Tinybird and have everything set and ready for tbtail to start sending logs to your account:

The last thing is installing tbtail. The details on how to do it are in its repo. We published deb and bin packages, and you can also compile it from source.

After it’s installed, you just need to run this command in the machine where the Nginx logs are created to send logs to Tinybird automatically.

Then, if you go to your dashboard, you’ll see a new pipe called query_grouped_requests. It defines an endpoint that lets you query your aggregated data. In the Pipe, you can click on the green “View API” button on the top right corner of the page to see its live documentation.