InfluxDB and Grafana

When buzzards are picking at your eyes, maybe it’s time to move a little. Do a little meandering, and you might discover that the world is larger, and more fun, than you imagined. Perhaps you realize that what was once a thriving oasis has now turned into a putrid cesspool riddled with parasites.

InfluxDB is what’s knowns as a “streaming database”. The idea is that it’s a database that collects samples over time. Once the sample is collected, it doesn’t change. Eventually the sample gets too old, and is discarded. This is different from traditional databases where the values may change over time, and the deletion of records is not normally based on age.

This sounds familiar doesn’t it?

Now, you probably want to draw some sort of timeline, or graph, that represents the values you popped into InfluxDB. Enter, Grafana. It’s a dashboard designer that can interface with InfluxDB (and other databases too) and show pretty graphs and tables in a web page w/o requiring any HTML/Javascript coding.

If you want to test this wonderful combination of software, you’ll probably want to run Docker, and visit this link.

Now, I’ve already abandoned the idea of using InfluxDB/Grafana for the kind of stuff I mess around with. InfluxDB’s strength is that it can return a condensed dataset over a potentially large time-range. And it can make fast and semi-complex computations over the samples it returns (usually of the statistical kind). But the kind of timeline information I usually record is not complex at all, and there aren’t really any additional calculations I can do over the data. E.g. what’s the average of “failed to connect” and “retention policy set to 10 days”.

InfluxDB is also schema-less. You don’t need to do any pre-configuration (other than creating your database), so if you suddenly feel the urge to create a table called “dunning” then you just insert some data into “dunning”. You don’t need to define columns or their types etc. you just insert data.

And you can do this via a standard HTTP call, so you can use curl on the command line, or use libcurl in your c++ app (which is what I did).

The idea that you can issue a single command to do a full install of InfluxDB and Grafana, and then have it consume data from your own little app in about the time it takes to ingest a cup of coffee says a lot about where we’re headed.

Contrast the “open platforms” that require you to sign an NDA, download SDKs, compile DLLs, test on 7 different versions of the server and still have to nurse it every time there’s a new version. Those systems will be around for a long time, but I think it’s safe to say they’re way past their prime.

 

 

Docker

I recently decided to take a closer look at Docker. Many of us have probably used virtualization at one point or other. I used Parallels on OSX some time ago to run Windows w/o having to reboot. Then there’s Virtualbox, VMWare, and Hyper-V that all allow you to run an OS inside another.

Docker does virtualization a bit differently.

The problem

When a piece of software has been created, it eventually needs to be deployed. If it’s an internal tool, the dev or a support tech will boot the machine, hunch over the keyboard and download and install a myriad of cryptic libraries and modules. Eventually, you’ll be able to start the wonderful new application the R&D dept. created.

Sometimes you don’t have that “luxury”, and you’ll need to create an automated (or at least semi-automated) installer. The end user will obtain it, usually via a download and then proceed to install it on their OS (often Windows) by running the setup.exe file.

The software often uses 3rd party services and libraries, that can cause the installer to balloon to several gigabytes. Sometimes the application needs a SQL server installation, but that, in turn, is dependent on some version of something else, and all of that junk needs to be packaged into the installer file – “just in case” the end user doesn’t have them installed.

A solution?

Docker offers something a bit more extreme: There’s no installer (other than docker). Once docker is running, you run a fully functional OS with all the dependencies installed. As absurd as this may sound, it actually leads to much smaller “installers” and it ensures that you don’t have issues with services that don’t play nice with each other. A Debian OS takes up 100Mbytes  (there’s a slim version at 55 Mbytes).

My only quip with docker is that it requires Hyper-V on windows (which means you need a Windows Pro install AFAIK). Once you install Hyper-V, it’s bye-bye Virtualbox. Fortunately, it is almost trivial to convert your old Virtualbox VM disks to Hyper-V.

There are a couple of things I wish were a bit easier to figure out. I wanted to get a hello-world type app running (a compiled app). This is relatively simple to do, once you know how.

How?

To start a container (with Windows as host), open PowerShell, and type

docker run -it [name of preferred OS]

You can find images on the docker hub, or just google them. If this is the first time you run and instance of the OS, docker will attempt to download the image. This can take some time (but once it is installed, it will spawn a new OS in a second or so). Eventually, you’ll then get a shell (that’s what the -it does), and you can use your usual setup commands (apt-get install … make folders and so on).

When you’re done, you can type “exit”

The next step is important, because the next time you enter “docker run -it [OS]” all your changes will have disappeared! To keep your changes, you need to commit them. To save your changes, start by entering this command:

docker ps -a

You’ll then see something like this

Take note of the Container ID, and enter

docker commit [Container ID] [image name]:[image tag]

For example

docker commit bbd096b363af debian_1/test:v1

If you take a look at the image list

docker images

You’ll see that debian_1/test:v1 is among them (along with the HDD use for that image).

You can now run the image with your changes by entering

docker run -it [image name]:[image tag]

You’ll have to do a commit every single time you want to save your change. This can be a blessing (like a giant OS based undo) and a curse (forgot to save before quitting), but it’s not something the end user would/should care about.

You can also save the docker image to a file, and copy it to another machine and run it there.

Granted, “Dockerization” and virtualization as a foundation is not for everyone, but for deployments that are handled by tech-savvy integrators it might be very suitable.

It sits somewhere between an appliance and traditional installed software. It’s not as simple as the former, not as problematic as the latter.