Jsonnet is the perfect solution for real-time monitoring with Ceph dashboards

Introduction

In the world of monitoring large-scale distributed systems, such as Ceph, having comprehensive and insightful dashboards is crucial for understanding the health and performance of the system.

One powerful tool used to generate these dashboards is Jsonnet, a superset of JSON that allows a flexible and dynamic dashboard creation by simplifying the JSON data.

In this blog post, I will show you how Jsonnet is used to build Ceph’s dashboards for monitoring.

Understanding Ceph Monitoring

Before we delve into the dashboard generation process, let’s first gain a basic understanding of Ceph monitoring. Ceph is a popular distributed storage system that provides reliable and scalable storage for various applications. To monitor the health and performance of a Ceph cluster, metrics such as cluster status, disk usage, network throughput, and latency need to be collected and visualized.

Dashboards

To monitor Ceph clusters, we use Grafana dashboards. They can be created manually, but since we need to set a lot of parameters we can consolidate them by using a JSON file that contains the following:

Metadata
- The JSON file begins with metadata that includes information about the dashboard, such as its title, description, and version. This metadata provides context and identification for the dashboard.
Panels
- Panels are the building blocks of a Grafana dashboard. Each panel represents a visual element on the dashboard, such as a graph, table, or singlestat (a single value visualization). The JSON file defines an array of panel objects, each with its own properties like title, type, and configuration specific to the visualization type.
Data Sources
- Grafana allows you to connect to various data sources to fetch and display metrics data. The JSON file contains information about the data sources used by the panels. This includes details such as the data source type (e.g., Prometheus, InfluxDB), connection settings, and query configurations.
Variables
- Variables provide a way to make dashboards more interactive and dynamic. They allow users to select different options or filter data based on certain criteria. The JSON file defines variables and their properties, such as name, label, data source, and available values.
Templating
- Templating enables parameterization of dashboards by substituting variables into queries and other configurations. The JSON file includes settings for templating, specifying the variables to be used, their query sources, and how they should be displayed.
Time Range
- Grafana allows users to adjust the time range for which data is displayed on the dashboard. The JSON file includes properties that define the default or preset time range options available to the user, such as “Last 1 hour,” “Last 7 days,” or custom ranges.
Annotations
- Annotations provide a way to mark specific events or points of interest on the dashboard. The JSON file can contain annotations configuration, including data sources, query settings, and appearance options.
Style and Layout
- The JSON file may include properties for defining the overall style and layout of the dashboard, such as the theme, color scheme, grid options, and sizing of panels.
Links and Navigation
- Grafana allows you to add navigation links to external resources or other dashboards. The JSON file can contain properties that define these links and their associated labels.

Understanding Jsonnet

What is Jsonnet

Jsonnet is a domain-specific language designed for generating JSON data. It provides a concise and flexible way to define complex JSON structures, making it an ideal choice for dashboard data.

Jsonnet extends JSON, adding variables, conditionals, functions, reusable components, and more to minimize duplication in defining configuration files. It’s got formatters, linters, and editor integrations.

Jsonnet?

According to the website,

The name Jsonnet is a portmanteau of JSON and sonnet, pronounced ‘jay sonnet’.

It was invented in 2014 at Google, and embodies years of experience configuring complex IT systems. Now, Jsonnet available to everyone. Thanks, Google.

Raise your hand if you just learned a new vocabulary word. (Portmanteau, no?)

By the way, credit for these examples goes to whoever wrote the Jsonnet doc. For this part, we will lean on their work. Kudos.

How it works

Let’s start with a simple example. We can declare a JSON file using an expressions and an object reference.

{
  person1: {
    name: 'Alice',
    welcome: 'Hello, ' + self.name + '!',
  },
  person2: self.person1 { name: 'Bob' },
}

This produces:

{
  "person1": {
    "name": "Alice",
    "welcome": "Hello, Alice!"
  },
  "person2": {
    "name": "Bob",
    "welcome": "Hello, Bob!"
  }
}

We can do the same thing using a function instead.

// A function that returns an object.
local Person(name='Alice') = {
  name: name,
  welcome: 'Hello, ' + name + '!',
};
{
  person1: Person(),
  person2: Person('Bob'),
}

Jsonnet turns this into:

{
  "person1": {
    "name": "Alice",
    "welcome": "Hello, Alice!"
  },
  "person2": {
    "name": "Bob",
    "welcome": "Hello, Bob!"
  }
}

Look familiar? Jsonnet is pretty flexible, giving you lots of options.

Now let’s turn up the complexity a little. We can define some variables for reuse throughout the configuration. Also, Jsonnet can write to many file formats, including .ini, .yaml, .sh, and .conf.

The following example names three output files to generate.

local application = 'my-app';
local module = 'uwsgi_module';
local dir = '/var/www';
local permission = 644;

{
  'uwsgi.ini': std.manifestIni({
    sections: {
      uwsgi: {
        module: module,
        pythonpath: dir,
        socket: dir + '/uwsgi.sock',
        'chmod-socket': permission,
        callable: application,
        logto: '/var/log/uwsgi/uwsgi.log',
      },
    },
  }),

  'init.sh': |||
    #!/usr/bin/env bash
    mkdir -p %(dir)s
    touch %(dir)s/initialized
    chmod %(perm)d %(dir)s/initialized
  ||| % {dir: dir, perm: permission},

  'cassandra.conf': std.manifestYamlDoc({
    cluster_name: application,
    seed_provider: [
      {
        class_name: 'SimpleSeedProvider',
        parameters: [{ seeds: '127.0.0.1' }],
      },
    ],
  }),
}

That Jsonnet file produces:

casandra.conf

"cluster_name": "my-app"
"seed_provider":
  - "class_name": "SimpleSeedProvider"
    "parameters":
      - "seeds": "127.0.0.1"

init.sh

#!/usr/bin/env bash
mkdir -p /var/www
touch /var/www/initialized
chmod 644 /var/www/initialized

uwsgi.ini

[uwsgi]
callable = my-app
chmod-socket = 644
logto = /var/log/uwsgi/uwsgi.log
module = uwsgi_module
pythonpath = /var/www
socket = /var/www/uwsgi.sock

That’s enough to understand Ceph monitoring and Jsonnet in general. Let’s get a preview of what’s coming next.

Part II and Part III, coming soon

In Part II, we will talk about Grafana and its amazing graphing capabilities. Grafana has an impressive set of Jsonnet libraries that support the builder pattern, making it easy to string together widgets and compose beautiful dashboards. So we will dive a little deeper into that.

Then we will wrap up this series by showing how all of this comes together in the latest code for the Ceph dashboard. That will be in Part III.

If you liked this post, you might want to try our Data Control Center. Sign up for a free trial of Koor software.