Hyperdistributed IP video

Verkada and Meraki is freaking everyone out. They’re selling direct, and they don’t respect the hierarchy. They’re seemingly attracting big $$$ from VC’s, and they’re landing million dollar deals at a fairly impressive clip.

These spry Silicon Valley kids with their typescript, Kubernetes, hoodies and sandals don’t give a damn about the gospel we’ve long established as the only truth under the sun.

The jury is still out on their approach. Right now, to me, it smells a lot like snake-oil. There’s a distinct aura of petroleum coming from the flask the energetic salesman is offering. I’ll let someone else drink the tonic and see how it goes before I’m willing to ingest this swill. But, make no mistake – they’re on to something, and they are probably right to taunt the rest of the industry.

So, why is the hyperdistributed model so good?

Let’s say you have a campus and need 500 cameras. With a traditional solution, those 500 cameras would be streaming – non-stop – to a central location (a server room usually). At the server room, you’d have a very expensive storage solution, typically with some sort of redundancy. Along with the storage, you’d need a few PC’s who’s main objective is to simply read from the camera, and store the feed for later review. A review that 99% of the time never happens. Video is compressed, sent across the network, stored on a disk, and then deleted when it gets too old. If the data-center goes down, it takes with it ALL recordings, and you’re basically blind. You can then install redundant data-centers (certainly not free!), which brings you maybe a little close to a distributed model.

It might be that you have an area that doesn’t have great WiFi network coverage, but you might have cellular access. Do you then establish a mesh-network all the way to the shed? Or do you let the camera stream 24/7 across the LTE network? Both are costly, and the cost is sunk if you later decide to move the camera.

I see two major problems with the hyperdistributed “Veraki” solutions: 1) if the camera is damaged, the evidence is gone, and 2) you’re currently locked in with either if you go that route.

The first problem is simple enough. Veraki simply offer a box that physically separates the imager from the storage via a cable, such that the recording can be better protected and will not disappear with the destruction of the camera.

The other is harder, because establishing a standard will almost immediately cause commoditization and ultimately loss of margins. I don’t think that’s a solvable problem, and the hypothetical risk is that someone actually establishes a standard for this sort of thing. Knowing the industry, that’s not going to happen in the next 10 years.

I don’t see the lack of camera setting, sync playback and so on as a long term problem. Currently, the idea is that all things should be web-based, but they’ll soon learn that the web was not made for video surveillance. If they’ve got a good database design, it shouldn’t be a problem to write a native app that could offer a more feature-rich experience for those who crave it. Spotify, for example, requires an installed app on the PC to play, the web site only offers account management.

If memory serves, Milestone actually worked on putting a lightweight version of their recorder inside cameras, and I think they signed some partners, but did it ever go anywhere? Such a solution would be strongly preferred, as “anyone” can communicate with a Milestone VMS and even write a client from scratch based on it.

Some companies will respond to this new threat by digging themselves deeper into a hole, while claiming to be reaching for the stars.

Looping Canned Video For Demos

Here’s a few simple(?) steps to stream pre-recorded video into your VMS.

First you need to install an RTMP server that can do RTMP to RTSP conversion. You can use Evostream, Wowza or possibly Nimblestreamer.  Nginx-rtmp won’t work as it does not support RTSP output.

Then get FFMpeg (windows users can get it here).

Find or create the canned video that you want to use, and store it somewhere accessible.

In this example, I have used a file called R1.mp4 and my RTMP server (Evostream) is located at 192.168.0.109. The command used is this:

ffmpeg -re -stream_loop -1 -i e:\downloads\r1.mp4 -c copy -fflags +genpts -f flv rtmp://192.168.0.109/live/r1

Once this is streaming (and you can verify using VLC and opening the RTMP url you provided), you can go to your VMS and add a generic RTSP camera.

For Evostream, the RTSP output is on a different port, and has a slightly different format, so in the recorder I add:

rtsp://192.168.0.109:5544/r1

Other RTMP servers may have a slightly different transform of the URL, so check the manual.

I now have a video looping into the VMS and I can run tests and benchmarks on the exact same feed w/o needing an IP camera.

 

 

Storage

Level1Techs explains a few things about storage. Worth a look.

Video Database Replication

Video Surveillance Databases are special. They are written to constantly, they are rarely read from, and the index is very simple (just a timestamp as the key). There’s no reason – really – to use anything fancy, certainly not SQL server.

I recently saw a marketing blurb for an expensive and cumbersome storage system that integrated to a VMS. It touted that the VMS had a “proprietary database highly optimized for video storage”. I guess “it uses the file system” did not sound fancy enough.

The entertaining puffery was uncovered as I was looking into the feasibility of geo-redundancy for a partner. Basically, they were looking for a fully mirrored backup system: If the primary site was to vanish, the backup site would take over, with all recorded data being readily available.

Database replication is nothing new; but typical database replication systems assume that you have a much higher outbound throughput than inbound. You may have a database with 2 million records, and if you add 1000 records per day, you’ll need those new records to propagate to the replication sets in your cluster – challenging, but a problem that has been solved a thousand times.

Video data is very different; its a constant torrent of data streaming into the system, and once in a while someone pulls out a few records to look at an incident. If the database uses the file system for its blocks, it’s almost trivial to provide replication. Just make sure the directory on the backup site looks identical to the one on the primary. This can be done with a simple rsync on Linux.

Another option is to use the Distributed Replicated Block Device (DRBD). This (Linux) tool allows you to create a drive that is mirrored 1:1 across a network. In other words, as files are written or changed, the exact same thing will happen on the backup drive. A Windows version appears to exist as well.

Surely, a better solution is to have the VMS be able to determine what files are most valuable, and push them to the remote site first. It might even chose to not mirror files that provide no value (zero motion files for example), or send a pruned version of the files to the backup system.

Depending on the sensitivity of the data, a customer might chose to extend/replicate their storage to the cloud. The problem here is that the upstream bandwidth is often limited, and thus in those cases a prioritization of the data is certainly needed.

Happy replicating…

 

 

Worlds Shittiest NVR pt. 3.

Our wonderful NVR is now basically a circular buffer in RAM, but we’d like to do a few things if motion (or other things) occur.

Many cameras support notification by email when things happen; while getting an email is nice enough, it’s not really what we want. Instead, we’ll (ab)use the mechanism as a way for the camera to notify our “NVR”.

First, we need a “fake” SMTP server, so that the camera will think that it is talking to a real one and attempt to send an actual email. When we receive the request to send the email we’ll simply do something else. An idea would be to move the temporary file on the RAM drive to permanent storage, but first, we’ll see if we can do the fake SMTP server in a few lines of code.

Start by downloading and installing node.js. Node.js allows us to run javascript code, and to tap into a vast library of modules that we can use via npm (used to stand for “Node Package Manager).

Assuming you’ve got node installed, we’ll open a command prompt and test that node is properly installed by entering this command:

node -v

You should now see the version number of node in the console window. If this worked, we can move on.

Let’s make a folder for our fake SMTP server first; Let’s pretend you’ve made a folder called c:\shittynvr. In the command prompt cd to that directory, and we’re ready to enter a few more commands.

We’re not going to write an entire fake SMTP server from scratch, instead, we’ll be using a library for node. The library is called simplesmtp. It is deprecated and has been superseded by something better, but it’ll work just fine for our purpose.

To get simplesmtp, we’ll enter this command in the prompt:

npm intall simplesmtp

You should see the console download some stuff and spew out some warnings and messages, we’ll ignore those for now.

We now have node.js and the simplesmtp library, and we’re now ready to create our “event server”.

Create a text file called “smtp.js”, add this code to the file, and save it.

const smtp = require ( "simplesmtp");
smtp.createSimpleServer({SMTPBanner:"My NVR"}, function(req){
  req.pipe(process.stdout);
  req.accept();
  
  // we can do other stuff here!!!

}).listen(6789);
console.log ( "ready" );

We can now start our SMTP server, by typing

node smtp.js

Windows may ask you if you want to allow the server to open a port, if you want your camera to send events to your PC, you’ll need to approve. If you are using a different firewall of some sort, you’ll need to allow incoming traffic on port 6789.

We should now be ready to receive events via SMTP.

The server will run as long as you keep the console window open, or until you hit CTRL+C to stop it and return to the prompt.

The next step is to set up the camera to send emails when things happen. When you enter the SMTP setup for your camera, you’ll need to enter the IP address of your PC and specify the port 6789. How you set up your camera to send events via email varies with manufacturers, so consult your manual.

Here’s an example of the output I get when I use a Hikvision camera. I’ve set it up so that it sends emails when someone tries to access the camera with the wrong credentials:

 Next time, we’ll look at moving files from temporary RAM storage to disk.

The Parts of an IP Camera

To understand where the IP camera market is headed, I think it’s important to understand how one of these things are put together.

Like most high tech devices, each product is really an amalgamation of parts from different manufacturers. In fact many products are the result of tight, but perhaps unappreciated, collaboration of several (sometimes competing) companies. I’d recommend listening to Freakonomics rundown of the “I, Pencil” essay (starts 7 minutes in).

So, an IP camera is not a pencil, but just like all pencil manufacturers don’t manufacture every single part of the pencil, but instead, they purchase the parts (graphite, brass, paint and so on) and every manufacturer puts the pencil together following roughly the same pattern.

And, so, when it comes to IP cameras, they too are composed of parts that are available to everyone who wants to start making cameras.

You’ll need a couple of things: A lens, a sensor, some circuitry and some code.

You’re not going to start making your own lenses or sensors, are you? Probably not, so you’ll get the lenses from a lens maker (and they may even outsource their manufacturing process even further), and the sensor from either Sony or Canon.

You’re not going to design your own CPU either (unless you’re Axis). Today, you’d be better off grabbing an ARM platform and use that to drive the sensor and interface. The other advantage is that ARM is well supported in the software world, so you’re already halfway there.

Now that you have the basics, you need to write some code to get it all working together. If you went the ARM route, it’s pretty simple to get a linux kernel running. Well.. “simple” is depends on your level of skill, but finding a few geeks who can do this shouldn’t take long. So you grab the Linux kernel, add Apache or perhaps GoAhead, you can add gStreamer too (do check the link, it is a great presentation by Axis) . The next thing you know, you have a jumble of cables and breadboards, burns on your fingers from the soldering iron, you haven’t seen your kids in 4 days and the smell is getting a little hard to stomach.

On top of that, you need to wrap this in an enclosure. There’s regulations to follow, tests that need to be carried out and so on. Then you have the nightmare of maintaining all those pieces of code, and trust me – if you wrote everything yourself, it would take even longer and be much harder to test and maintain.

What if there was a company, that could do all of the above? And just stick my name on the box? After all, my company would pick the same lens, the same sensor, the same board and the same software, so why not do it?

I have no intention of starting production of a Raspberry Pi Zero based IP camera, but I know that I can make one for ~$40 (and that’s buying all the parts retail). Not only will this thing work as an IP camera, it can work as a full fledged stand-alone VMS.

In other words, the question is: if some washed up coder in Copenhagen can build a fully functional “IP camera” for $40, I think you’re going to face a tough time if you’ve based your entire organization around selling your cheapest cameras for $250+ (they may be “even more good enough”, but who cares?).

Obviously, my camera is not going to be materially different from the other guy’s cameras. We’re all going to use the same bits and pieces, including software, even the damn protocols are going to be the same.

So, I think we’re going to see a race to the bottom in terms of prices. The cameras will look and perform almost identically across brands, use the same protocols, and be completely interchangeable, much to the chagrin of the incumbents, so the USP for the brands in this realm will have be something else.

VMS Software, perhaps…

 

 

 

 

 

 

Ryzen

Youtubers are disappointed with Ryzen. They expected it to crush Intel in every single benchmark, and I had hoped that it would too. What was I thinking?

The problem that AMD has is that a lot of people seem think that you can get a baby faster if you just add more women.

I’ve been watching a lot of indie coders do a lot of single loop coding, which obviously will not scale across several cores. They are obsessed with avoiding L1/L2 cache misses, which is fine, but at the same time, they rarely talk about multi-threading. Some of the benchmarks I have seen leaves the CPU at 30% utilization across all cores, which means that there’s a lot of untapped potential.

Games with lots of autonomous and complex entities should scale quite well as – if the coder is not spending all his time on organizing his data in a way that makes little sense on a multi-core system, and is willing to shed the dogma that threads are bad.

I am not replacing my 3770K though. I was very tempted to get something that could substantially increase compilation times, but I spend <1% on compilations, so even a massive improvement in compilations would not really improve my productivity overall. And I am not thrilled on having to buy new RAM once again…

 

TileMill and Ocularis

A long, long time ago, I discovered TileMill. It’s an app that lets you import GIS data, style the map and create a tile-pyramid, much like the tile pyramids used in Ocularis for maps.

There are 2 ways to export the map:

• Huge JPEG or PNG
• MBTiles format

So far, the only supported mechanism of getting maps into Ocularis is via a huge image, which is then decomposed into a tile pyramid.

Ocularis reads the map tiles the same way Google Maps (and most other mapping apps) reads the tiles. It simply asks for the tile at x,y,z and the server then returns the tile at that location.

We’ve been able to import Google Map tiles since 2010, but we never released it for a few reasons:

• Buildings with multiple levels
• Maps that are not geospatially accurate (subway maps for example)
• Most maps in Ocularis are floor plans, going through google maps is an unnecessary extra step
• Reliance on an external server
• Licensing
• Feature creep

If the app is relying on Google’s servers to provide the tiles, and your internet connection is slow, or perhaps goes offline, then you lose your mapping capability. To avoid this, we cache a lot of the tiles. This is very close to bulk download which is not allowed. In fact, at one point I downloaded many thousands of tiles, which caused our IP to get blocked on Google Maps for 24 hours.

Using MBTiles

Over the weekend I brought back TileMill, and decided to take a look at the MBTile format. It’s basically a SQLite DB file, with each tile stored as a BLOB. Very simple stuff, but how do I serve the individual tiles over HTTP so that Ocularis can use them?

Turns out, Node.js is the perfect tool for this sort of thing.

Creating a HTTP server is trivial, and opening a SQLite database file is just a couple of lines. So with less than 50 lines of code, I had made myself a MBTile server that would work with Ocularis.

A few caveats : Ocularis has the Y axis pointing down, while MBTiles have the Y axis pointing up. Flipping the Y axis is simple. Ocularis has the highest resolution layer at layer 0, MBTiles have that inverted, so the “world tile” is always layer 0.

So with a few minor changes, this is what I have.

 

I think it would be trivial to add support for ESRI tile servers, but I don’t really know if this belongs in a VMS client. The question is time was not better utilized by making it easy for the GIS guys to add video capabilities to their app, rather than having the VMS client attempt to be a GIS planning tool.

 

NUUO

First Axis, now NUUO. Full disclosure.

Razor Software

It dawned on me that the VMS software, in many cases, are equivalent to razors… or pizzas, or burgers. It’s reaching the point where it’s really hard to sell anything better than just “good enough”. Sure, there are 5-blade razors with gels to soothe the skin, and there are “gourmet” burgers and pizzas (I still prefer Wendy’s, and I’ve been to a lot of burger places). So unless you can sell some sort of “experience”, when peddling easily fungible goods, you’ll be competing on price.

I recently was on vacation in Greece and I noticed that the cash register software was (apparently) DOS based, and hadn’t been updated since 2004. The service was as fast as anywhere else, and the software had no issue looking up the price of an item as quickly as the operator could scan the barcode or manually key in the item. A local hardware store here in Copenhagen has a similar system, and the Luddite discount supermarket chain, Aldi, didn’t have barcode scanners until recently.

In the good old days (of scurvy and gout), the store would simply write down what was sold, and at what price in a little black book. Back then, I doubt there were price tags on items, as it was the Quakers who introduced fixed prices.

Cash register by the National Cash Register Co., Dayton, Ohio, United States, 1915.

The first register came about when saloon owner, James Ritty got fed up with his employees stealing from him, so he invented the cash register. A contraption he called the “Ritty’s Incorruptible Cashier”. Soon after, the invention was sold to the company that later became NCR.

NCR went on to add bookkeeping features to the cash register, so not only would you prevent the employees from stealing (it’s intended purpose), but it would also make your life easier as a shop-keeper.

Electronic cash registers was the next step. Clearly, going from a mechanical contraption to an electronic version made sense. It was cheaper, more reliable, faster and had more features.

Then came the DOS based POS systems that offered yet more flexibility, were easier to use and could interface with more printers and cash drawers.

Today, the local hipster hangout coffee shop down the street is now using an iPad with a credit card scanner attached.

As I see it, the embedded electronic register is equivalent to a DVR with some NTSC cameras. The DOS based register corresponds to the windows based IP video recorders that are fairly common today. And finally, a plug’n play video surveillance solution in the DropCam vein is peered to the iPad POS software.

My thinking is this: A tool that solves a particular, well defined problem, won’t be upgraded, at least not until the new tool offers a substantially better ROI. Adding bookkeeping to the mechanical cash made sense, but I doubt people replaced their old working mechanical registers with the slightly improved variant. New customers will buy the feature-rich variant if the price points are similar, but it’s very hard to change a substantial premium for features that really aren’t needed. And even harder to convince people to replace a working system with something more expensive, but only offers marginally useful features. I suspect that the reason I am seeing a lot of stores with DOS (or DOS-like) registers, is because the functionality is simply good enough, and there’s no meaningful ROI on upgrading to a, say, Windows 10 or iOS version.

And that brings me to VMS systems. Clearly, the analog systems were a pain in the ass. Tapes that were worn out, rendering the system useless when something did happen. Terrible low light performance and so on. The DVR systems were substantially better, causing people to upgrade. And for a while, IP based cameras were very, very expensive, but as prices came down, they started to offer a substantially better ROI than DVRs (in many respects, but not all).

The key point here is “good enough”: I don’t think we are quite there yet, but eventually we will arrive at a point where all VMS/NVRs will be pretty much plug and play. It’ll be a piece of cake to replace a camera (why the hell is this still a challenge on some VMSs?), hopefully no-one is going to spend hours tweaking resolutions, messing around with VBR and CBR, or mess around with “motion sensitivity” settings.

If I had my way, I’d let the “other guys” come up with new condiments, and ways to serve their turd-sandwiches, and instead offer cupcakes. Some people don’t like cupcakes, some prefer donuts, but I bet you that most people would pick a cupcake over any turd-sandwich. There are always people that complain, monday-morning-quarterbacks (some made a career of it), who demand some weird condiment that comes with turd-sandwich #2, and in some cases, sinking your teeth into a turd-sandwich is simply the only way to achieve some narrow goal.

I bet that once someone offers a cupcake, it’s game over for the turd-sandwich stores.