Video Database Replication

Video Surveillance Databases are special. They are written to constantly, they are rarely read from, and the index is very simple (just a timestamp as the key). There’s no reason – really – to use anything fancy, certainly not SQL server.

I recently saw a marketing blurb for an expensive and cumbersome storage system that integrated to a VMS. It touted that the VMS had a “proprietary database highly optimized for video storage”. I guess “it uses the file system” did not sound fancy enough.

The entertaining puffery was uncovered as I was looking into the feasibility of geo-redundancy for a partner. Basically, they were looking for a fully mirrored backup system: If the primary site was to vanish, the backup site would take over, with all recorded data being readily available.

Database replication is nothing new; but typical database replication systems assume that you have a much higher outbound throughput than inbound. You may have a database with 2 million records, and if you add 1000 records per day, you’ll need those new records to propagate to the replication sets in your cluster – challenging, but a problem that has been solved a thousand times.

Video data is very different; its a constant torrent of data streaming into the system, and once in a while someone pulls out a few records to look at an incident. If the database uses the file system for its blocks, it’s almost trivial to provide replication. Just make sure the directory on the backup site looks identical to the one on the primary. This can be done with a simple rsync on Linux.

Another option is to use the Distributed Replicated Block Device (DRBD). This (Linux) tool allows you to create a drive that is mirrored 1:1 across a network. In other words, as files are written or changed, the exact same thing will happen on the backup drive. A Windows version appears to exist as well.

Surely, a better solution is to have the VMS be able to determine what files are most valuable, and push them to the remote site first. It might even chose to not mirror files that provide no value (zero motion files for example), or send a pruned version of the files to the backup system.

Depending on the sensitivity of the data, a customer might chose to extend/replicate their storage to the cloud. The problem here is that the upstream bandwidth is often limited, and thus in those cases a prioritization of the data is certainly needed.

Happy replicating…

 

 

Advertisements

MxPEG to H.264

Get a Raspberry Pi, or one of the very cheap clones. Then install FFMpeg and an RTMP server with RTSP capability (EvoStream, Wowza).

Make sure the RTMP server is operational.

Ask FFMpeg to convert from MxPEG to H.264 and broadcast to the RTMP server, by entering this command (on one line)

ffmpeg 
  -f mxg 
  -i http://[user]:[pass]@[camera-host]/cgi-bin/faststream.jpg?stream=mxpeg 
  -codec:v libx264 
  -b:v 500k 
  -maxrate 500k 
  -bufsize 1000k 
  -vf scale=-1:480 
  -threads 0 
  -an 
  -f flv [rtmp address]

If you are using EvoStream, you might have entered something like this for the RTMP address:

rtmp://[ip of evostream]/live/mobotix

If that is the case, you can add a generic RTSP camera to your VMS

rtsp://[ip of evostream]/mobotix

The MxPEG stream will now be converted to H.264 and recorded as such. You’ll miss out on the advantages of MxPEG, but sometimes there’s no other way around the issue.

Camera Proxy

There’s a lot of paranoia in the industry right now, some warranted, some not. The primary issue is that when you plug something into your network you basically have to trust the vendor to not spy on you “by design” and to not provide a trivial attack vector to 3rd parties.

First things first. Perhaps you remember that CCTV means Closed Circuit Television. Pay attention to those first two words. I am pretty sure 50% or more of all “CCTV” installations are not closed at all. If your CCTV system is truly closed, there’s no way for the camera to “call home”, and it is impossible for hackers to exploit any attack vectors because there’s no access from the outside world to the camera. There are plenty of PC’s running terrible and vulnerable software out there, but as long as these systems are closed, there’s no problem. Granted, it also limits the flexibility of the system. But that’s the price you pay for security.

In the opposite end of the spectrum are cameras that are directly exposed to the internet. This is a very bad idea, and most professionals probably don’t do that. Well… some clearly do, because a quick scan of the usual sites reveal plenty of seemingly professional installations where cameras are directly accessible from the internet.

To expose a camera directly to the internet you usually have to alter the NAT tables in your router/firewall. This can be a pain in the ass for most people, so another approach is used called hole-punching. This requires a STUN server between the client sitting outside the LAN (perhaps on an LTE connection via AT&T) and the camera inside the LAN. The camera will register with the STUN server via an outbound connection. Almost all routers/firewalls allow outbound connections. The way STUN servers work, probably confuse some people, and they freak out when they see the camera making a connection to “suspicious” IP but that’s simply how things work, and not a cause for alarm.

Now, say you want to record the cameras in your LAN on a machine outside your LAN, perhaps you want an Azure VM to record the video, but how will the recorder on Azure (outside your LAN) get access to your cameras that are inside the LAN unless you set up NAT and thus expose your cameras directly to the internet?

This is where the $10 camera proxy comes in (the actual cost is higher because you’ll need an SD card and a PSU as well).

So, here’s a rough sketch of how you can do things.

  • On Azure you install your favorite VMS
  • Install Wowza or EvoStream as well

EvoStream can receive an incoming RTMP stream, and make the stream available via RTSP, it basically changes the protocol, but uses the same video packets (no transcoding). So, if you were to publish a stream at say rtmp://evostreamserver/live/mycamera, that stream will be available at rtsp://evostreamserver/mycamera. You can then add a generic RTSP camera that reads from rtsp://evostreamserver/mycamera to your VMS.

The next step is to install the proxy, you can use a very cheap Pi clone, or a regular PC.

  • Determine the RTSP address of the camera in question
  • Download FFMpeg
  • Set up FFMpeg so that it publishes the camera to EvoStream (or Wowza) on Azure

Say you have a camera that streams via rtsp://192.168.0.100/video/channels/1, the command looks something like this (all on one line)

ffmpeg -i rtsp://username:password@192.168.0.100/video/channels/1 
-vcodec copy -f flv rtmp://evostreamserver/live/mycamera

This will make your PC grab the AV from the camera and publish it to the evostream server on Azure, but the camera is not directly exposed to the internet. The PC acts as a gateway, and it only creates an outbound connection to another PC that you control as well.

You can now access the video from the VMS on Azure, and your cameras are not exposed at all, so regardless how vulnerable they are, they will not expose any attack vectors to the outside world.

Using Azure is just an example, the point is that you want to isolate the cameras from the outside world, and this can be trivially accomplished by using a proxy.

As a side note. If cameras were deliberately spying on their users, by design, this would quickly be discovered and published. That there are bugs and vulnerabilities in firmware is just a fact of life and not proof of anything nefarious, so calm down, but take the necessary precautions.

Orange Pi One

This is getting ridiculous.

I just received my $10 computer from China. I paid a premium for the (required) SD card as I do not have the patience to wait for one to arrive in the mail. My 5V/2A charger for my old, functional, PSP works as a power supply. I then downloaded Armbian and booted.

A few commands later, and I have a $20 dollar camera proxy.

I don’t actually plan to use it as my camera proxy, but as a small controller for a number of sensors I plan to add. For example using a cheap modified PIR sensor as input to the controller.

As you may know, I also have a Raspberry Pi 2. This little device is incredibly stable, and has only been rebooted once in the last 3 months, and that was by accident.

Hopefully you’ll be able to get a $100 device that you simply plug into your infrastructure, and that little device will work as standalone, or as a node in a much larger VMS, but that’s a bigger project that I might pick up later.

Some of the commands I used :

sudo apt-get update

sudo apt-get install ffmpeg

ffmpeg -i rtsp://...... -vcodec copy -f flv rtmp://....

Stop Crap Shooting, Blindfolded, In The Dark

Most people add some sort of tracker to their website. It allows you to monitor the performance of your posts, your geographical reach, what articles made a splash and so on. You instinctively know that you need to measure, if you want to know where to focus your attention. Talking to people is one way of “measuring”, but it is notoriously unreliable; an old saying is that “people bitch about the stuff they use”. So when people “just love the new feature” it usually means they are not using it.

Eqatec is a tracker for .NET applications, which works almost like Google analytics for web-pages. There are privacy concerns, off course, and for our vertical it is even more sensitive. But for beta-testing this is simply a must (naturally client consent is mandatory), it takes a lot of the guesswork and – frankly – BS – out of the equation. The beta testers provide valuable input, but unless the input is anchored in REAL use, the input is meaningless. Some people will balk at the notion of being tracked (yet use google on a daily basis), while others see no problem at all. I am probably in the latter group. In fact my radio-listening habits are being monitored almost 24/7 (all death-metal all the time 😉 ). The polling agency ASKS, but also measures what I am listening to – let’s see if the two match up.

Linkage made a great video about it. See it here.

NVR Integrators Toolbox

I never realized the importance of configuration tools – until now! I suppose as a developer I never really considered the difficulty of designing a complete video surveillance installation, but relied on the old adage “when in doubt, add machines”. But where should I place the cameras, how much data will they record and so on is a big part of these questions, and we only provide relatively simple online “calculators” that certainly does not help you visualize the entire installation.

I guess I still have a lot to learn (even after 10 years in the business 🙂 )