Docker

I recently decided to take a closer look at Docker. Many of us have probably used virtualization at one point or other. I used Parallels on OSX some time ago to run Windows w/o having to reboot. Then there’s Virtualbox, VMWare, and Hyper-V that all allow you to run an OS inside another.

Docker does virtualization a bit differently.

The problem

When a piece of software has been created, it eventually needs to be deployed. If it’s an internal tool, the dev or a support tech will boot the machine, hunch over the keyboard and download and install a myriad of cryptic libraries and modules. Eventually, you’ll be able to start the wonderful new application the R&D dept. created.

Sometimes you don’t have that “luxury”, and you’ll need to create an automated (or at least semi-automated) installer. The end user will obtain it, usually via a download and then proceed to install it on their OS (often Windows) by running the setup.exe file.

The software often uses 3rd party services and libraries, that can cause the installer to balloon to several gigabytes. Sometimes the application needs a SQL server installation, but that, in turn, is dependent on some version of something else, and all of that junk needs to be packaged into the installer file – “just in case” the end user doesn’t have them installed.

A solution?

Docker offers something a bit more extreme: There’s no installer (other than docker). Once docker is running, you run a fully functional OS with all the dependencies installed. As absurd as this may sound, it actually leads to much smaller “installers” and it ensures that you don’t have issues with services that don’t play nice with each other. A Debian OS takes up 100Mbytes  (there’s a slim version at 55 Mbytes).

My only quip with docker is that it requires Hyper-V on windows (which means you need a Windows Pro install AFAIK). Once you install Hyper-V, it’s bye-bye Virtualbox. Fortunately, it is almost trivial to convert your old Virtualbox VM disks to Hyper-V.

There are a couple of things I wish were a bit easier to figure out. I wanted to get a hello-world type app running (a compiled app). This is relatively simple to do, once you know how.

How?

To start a container (with Windows as host), open PowerShell, and type

docker run -it [name of preferred OS]

You can find images on the docker hub, or just google them. If this is the first time you run and instance of the OS, docker will attempt to download the image. This can take some time (but once it is installed, it will spawn a new OS in a second or so). Eventually, you’ll then get a shell (that’s what the -it does), and you can use your usual setup commands (apt-get install … make folders and so on).

When you’re done, you can type “exit”

The next step is important, because the next time you enter “docker run -it [OS]” all your changes will have disappeared! To keep your changes, you need to commit them. To save your changes, start by entering this command:

docker ps -a

You’ll then see something like this

docker

Take note of the Container ID, and enter

docker commit [Container ID] [image name]:[image tag]

For example

docker commit bbd096b363af debian_1/test:v1

If you take a look at the image list

docker images

You’ll see that debian_1/test:v1 is among them (along with the HDD use for that image).

You can now run the image with your changes by entering

docker run -it [image name]:[image tag]

You’ll have to do a commit every single time you want to save your change. This can be a blessing (like a giant OS based undo) and a curse (forgot to save before quitting), but it’s not something the end user would/should care about.

You can also save the docker image to a file, and copy it to another machine and run it there.

Granted, “Dockerization” and virtualization as a foundation is not for everyone, but for deployments that are handled by tech-savvy integrators it might be very suitable.

It sits somewhere between an appliance and traditional installed software. It’s not as simple as the former, not as problematic as the latter.

Advertisements

Safe, Easy, Advanced

You can only pick 2 though.

Admitting mistakes is hard; it’s so hard that people will pay good money just to be told that they are not to blame for the mistake. That someone else is responsible for their stupidity. And sometimes they’re right, sometimes not.

Anton Yelchin was only 27 when he died, he left his car in neutral on an incline. The car started rolling and killed him. Since it would be unbearable to accept that Anton simply made a mistake, lawsuits were filed.

Another suitor claimed that the lever was too complex for people to operate, therefore the manufacturer is liable for the damage that occurs when people don’t operate them correctly. The car had rolled over her foot, and while there were no broken bones, she was now experienced “escalating pains”, and demanded reparations. One argument was that the car did not have the same feature as a more expensive BMW.

Tragically, every year more than 30 kids are forgotten in cars and die. When I bring this up with people, everyone says “it won’t ever happen to us”, and so there’s zero motivation to spend extra on such a precaution. The manufacturers know this, and since there’s also liability risk, they are not offering it. So, every year, kids bake to death in cars. It’s a gruesome fate for the kids, but the parents will never recover either.

Is it wrong to “blame the victim”?

I think the word “blame” has too many negative connotations associated to be useful in this context. Did the person’s action/inaction cause the outcome? If the answer is a resounding yes, then sure…  we can say that we “blame the victim”.

It’s obviously a gray area. If a car manufacturer decides that P should mean neutral and N should mean park, and writes about this in their manual and tells the customers as they sign the contract, then I wouldn’t blame an operator for making the mistake. The question is – would a person of “normal intelligence” be more than likely to make the same mistake?

In our industry, not only are we moving the yard-post of what “normal intelligence” means. Some of the most hysterical actors are using the bad practices of the layman and arguing that the equipment, therefore, can’t be used by professionals.

It feels like it’s entirely reasonable to argue no-one should drive 18-wheelers because random people bought modified trucks at suspect prices in a grey market and then went ahead and caused problems for a lot of people.

As professionals, we’re expected to have “higher intelligence” when it comes to handling the equipment. You can’t call yourself “professional” if you have to rely on some hack and his gang to educate you online or through their “university”. And you sure as hell can’t dismiss the usability of a device based on what random amateurs do with it.

So what gives? You have a bunch of people who act like amateurs but feel like “professionals” because they are paying good money for this industry’s equivalent of 4chan and got paid to install a few units here and there.

It seems to me that the hysterical chicken-littles of this industry are conflating their own audiences with what actual professionals are experiencing. E.g. if someone suggests using a non-standard port to “protect their installation”, then you know that the guy is not professional (doesn’t mean he’s not paid, just means he’s not competent).

And that’s at the core of this debacle: people that are incompetent, feel entitled to be called professionals, and when they make mistakes that pros would never make, it’s the fault of the equipment and it’s not suitable for professionals either.

So, as I’ve stated numerous times, I have a Hikvision and an Axis camera sitting here on my desk. Both have default admin passwords, yet I have not been the victim of any hack – ever. The Hikvision camera has decent optics (for a surveillance camera) and provides an acceptable image at a much lower cost than the “more secure” option.

And I’ll agree that getting video from that camera to my phone, sans VPN is not “easy” for the layman. But it doesn’t have to be. It just has to be easy for the thousands of competent integrators know what to do, and more importantly, what not to do.

That said; the PoC of the HikVision authentication bypass string should cause heads to roll at Hikvisions (and Dahuas) R&D department. Either there’s no code-review (bad) or there was, and they ignored it (even worse). There’s just no excuse for that kind of crap to be present in the code. Certainly not at this day and age.

 

Worldwide Hack

Cameras have vulnerabilities, some easier to exploit than others. Unless you have some sort of mental defect, this is hardly news. This old fart wrote about it in 2013/2014, but it still affects a lot of people..

If you’re a bit slow in the head, you might want to take your hard earned cash, and give it to some sociopathic megalomaniac who thinks he’s the savior of the world, and feel helpless and vulnerable as you cower under the threat of the “big unknown”.

A recent hyped headlines exclaims:

“WORLDWIDE HACK”

But, you know, with this new-fangled internet, it’s pretty easy to do something “worldwide”; any script kiddie in their mother’s basement can hit every single IP that is exposed to the internet if they want. “Worldwide” don’t mean diddly squat these days. Unless you’re living in the 80’s, desperately trying to get your damn VCR fixed, so you can watch those old tapes you kept.

Now, Cameras, NVRs and DVRs with shitty security, straight to the internet? Bad fucking idea. Doesn’t mean that people don’t do it. Like drinking 2 gallons of Coke and wolfing down junk food for lunch and dinner is a bad idea -yet millions of people (actually worldwide) do it.

So you can make an easy buck selling subscriptions that places the blame squarely on the coke and pizza for the obesity epidemic. After all, who doesn’t like to be absolved of their sins, and pointing the finger at everyone else.  “The magazine says I am not to blame”, and then you can continue your gula uninhibited.

A wise person would not expect Coke or Papa Johns to spend millions of dollars showing the bad effects of poor dietary choices. They’ll continue to show fit girls and boys enjoying a coke and pizza responsibly, but the bulk of their income is certainly not derived from people with a BMI < 20.

While I understand the desire to believe that “easy” equates “correct”, it never ceases to amaze me that people don’t take any precautions. Maybe my mistake is that I am underestimating how gullible people really are (and my sociopath nemesis isn’t).

While this big, nasty, “worldwide” attack is taking place, I still haven’t seen anyone hack my trusty old Hikvision camera sitting here on my desk… must be a coincidence that I wasn’t hit.

Debtors Prison

There’s a wonderful term called “technical debt”. It’s what you accrue when you make dumb mistakes, and instead of correcting the mistake, and taking the hit up front, you take out a small loan, patch up the crap with spittle and cardboard, and ship the product.

kid_credit
Yay! Free money!!!

Outside R&D technical debt doesn’t seem to matter. It’s like taking your family to a restaurant and racking up more debt; the kids don’t care, to them, the little credit card is a magical piece of plastic, and the kids are wondering why you don’t use it more often. If they had the card, it would be new PlayStations and drones every day.

Technical debt is a product killer; as the competition heats up, the company wants to “rev the engine”, but all the hacks and quick fixes mean that as soon as you step on the gas, the damn thing falls apart. The gunk and duct tape that gave you a small lead out of the gate, but in the long run, the weight of all that debt will catch up. It’s like a car that does 0-60 in 3 seconds but then dies after 1 mile of racing. Sure it might enter the race again, limp along for a few rounds, then back to the garage, until it eventually gives up and drops out.

Duct Tape Car Fix - 03
Might get you home, but you won’t win the race with this fix

Why does this happen?

A company may masquerade as a software company and simply pile more and more resources into “just fix it” and “we need” tasks that ignore the real need to properly replace the intake pipe shown above. “If it works, why are you replacing it”, the suit will ask, “my customer needs a sunroof, and you’re wasting time on fixing something that already works!”.

So, it’s probably wise to look at the circumstances that caused the company to take on the debt in the first place. An actual software company might take technical debt very seriously, and very early on they will schedule time for 3 distinct tasks:

  1. Ongoing development of the existing product (warts and all),
  2. Continued re-architecting and refactoring of modules,
  3. Development of the next generation product/platform

Any given team (dependent on size, competency, motivation, and guidance) will be able to deliver some amount of work X. The company sells a solution that requires the work Y. Given that Y < X, the difference can be spent on #2 and #3. The bigger the difference, the better the quality of subsequent releases of the product. If the difference is small, then (absent team changes), the product will stagnate. If Y > X then the product will not fulfill the expectations of the customer. To bridge the gap until the team can deliver an X > Y, you might take on some “bridge debt”. But if the bridge debt is perpetual (Y always grows as fast or faster than X), then you’re in trouble. If Y > X for too long, then X might actually shrink as well, which is a really bad sign.

Proper software architecture is designed so that when more (competent) manpower is added, X grows. Poor architecture can lead to the opposite result. And naturally, incompetent maintenance of the architecture itself (an inevitable result of a quick-fix culture), will eventually lead to the problematic situation where adding people lead to lower throughput.

A different kind of “debt” is the inability to properly value the IP you’ve developed. The cost of development is very different from the value of the outcome. E.g. a company may spend thousands of hours developing a custom log handler, but the value of such a thing is probably very low. This is hard to accept for the people involved, and it often leads to friction when someone points out that the outcome of 1000 hours of work is actually worthless (or possibly even provides a net negative value for the product). A lot of (additional) time may be spent trying to persuade ourselves that we didn’t just flush 1000 hours down the drain, as we’re more inclined to believe a soothing lie than the painful truth.

Solutions?

A company that wants to solve the debt problem must first take a good look at its core values. Not the values it pretends to have, but the actual values; what makes management smile and how it handles the information given to them. Does management frown when a scalability issue is discovered, do they yell and slam doors, points out 20 times that “we will lose the customer if we don’t fix this now!”. The team lead hurries down the hallway, and the team pulls out cans of Pringles and the start ripping off pieces of tape.

The behavior might make the manager feel good. The chest-beating alpha-manager put those damn developers in their place, and got this shit done!. However, over the long run, it will lead to 3 things : 1) Developers will do a “quick fix”, because management wants this fixed quickly, rather than correctly, 2) Developers will stop providing “bad news”, and 3) developers that value correctness and quality will leave.

To the manager, the “quality developer” is not an asset at all. It’s just someone who wants to delay everything to fix an intake that is already working “perfectly”. So over time, the company will get more and more duct-tapers and hacks, and fewer craftsmen and artisans.

The only good thing about technical debt (for a coder) is that it belongs to the company, and not to the employees. Once they’re gone, they don’t have to worry about it anymore. Those that remain do, and they now have to work even harder to pay it back.

debt_mountain2

Technical Innovation

This might be wonky, so if you’re not into that sort of thing, you can save yourself 5 minutes and skip this one.

Enterprise installations of software usually consists of several semi-autonomous clusters (or sites) that are arranged in either some sort of hierarchy. Usually a tree-like structure, but it could be a mesh or graph.

Each cluster may have hundreds or even thousands of sensors (not just cameras) that all can be described by their state. A camera may be online, it may be recording, a PIR might be triggered or not and so on. Basically, sensors have three types of properties: Binary, scalar and complex ones (an LPR reader that returns tag and color of a car).

The bandwidth inside each cluster is usually decent (~GBit), and if it isn’t an upgrade of the network is usually a one time fee. Sure, there might be sensors that are at remote locations where the bandwidth is low, but in most cases bandwidth is not an issue inside the cluster. Unfortunately, the bandwidth from the cluster to the outside world is usually not as ample.

It’s not uncommon that a user at the top of the hierarchy will see a number of clusters, and it seems reasonable that this user also wants to see the state of each of the sensors inside the cluster. This, then requires that the state information of each sensor is relayed from the cluster to the client. The client may have a 100 Mbit downstream connection, but if the upstream path from the cluster to the client is a mere 256 Kbit/s then getting the status data across in real time is a real problem.

In the coding world there’s an obsession with statelessness. Relying on states is hard, so is deleting memory after you’ve allocated it, and debugging binary protocols isn’t cakewalk either, so programmers (like me) try to avoid these things. We have garbage collection (which I hate), we have wildly verbose and wasteful containers such as SOAP (which I hate) and we have polling (which I hate).

So, let’s consider a stateless design with a verbose container (but a lot more terse than soap): The client will periodically request the status.

client:
  <request>
    <type>status</type>
    <match>all</match>
  </request>

server:
  <response>
     <sensor id="long_winded_guid_as_text" type="redundant_type_info">
        <state>ON</state>
     </sensor>
  </response>

The client request can be optimized, but it’s the server that is having the biggest problem. If we look at the actual info in the packet, we are told that a sensor, with a guid, is ON.

Let’s examine the entropy of this message.

The amount of data needed to specify the sensor depends heavily on the number of sensors in the system. If there’s just 2 sensors in total, then the identifier requires 1 bit of information (0 or 1), if there are 4 sensors, you’ll need 2 bits (00, 01, 10 and 11), and so if there are 65000 sensors, you’ll need 16 bits (2 bytes).

The state is binary, so you’ll need just a single bit to encode the state (0 for off, 1 for on).

However, since the state information can be of 3 different types, you’ll need a status type identifier as well. However, this information only needs to be encoded in the message if the type actually changes. If the status type stays the same, it can be cached on the receiving side. However, in order to remain stateless, we’re going to need 2 bits to encode the type (00 for binary, 01 for scalar, 10 for complex).

so, for a system with 60.000 sensors, a message might look like this

[ sensor id          ] [ type ] [ value ]
  0000 0000 0000 0001    00       1

19 bits in total

The wasteful message is 142 bytes long, or 1136 bits, or 59 times larger, and it is actually pretty terse compared to some of the stuff you’ll see in the real world. In terms of bandwidth, the compact and “archaic” binary protocol can push 59 times as many messages through the same pipe!

Now, what happens if we remove the statelessness? We could cache the status type on the receiving side, so we’d get down to 17 bits (a decent improvement), but we could also decide to not poll, and instead let the server push data only when things change.

The savings here are entirely dependent on the configuration of the system, and sensors with scalar values probably have to continuously send the value. A technique used when recording GPS info is to only let the device send its position when it has changed more than a certain threshold or when a predefined duration has passed. That means that if you’re stationary, we only have location samples at a 1 sample per minute for example, but as you move, the sample rate increases to 1 per second. The same could be used for temperature, speed and other scalar values.

So, how often does a camera transition from detecting some motion, to not detecting any? Once per second? Once per minute? Once per hour? Probably all 3 over a 24 hour period, but on average, let’s say you have one transition every 5 minutes. If you’re running a 1 Hz polling rate vs. a sensor that reports state changes only, you’re looking at a 300x reduction in bandwidth.

In theory we’re now about 17000 times more efficient using states and binary protocols.

In practice, the TCP overhead makes quite a bit less. We might also add a bit of goo to the binary protocol to make it easier to parse, and that’s the other area where things improve: processing speed.

Let’s loosen up the binary protocol a bit by byte-aligning each element. This increases the payload to 32 bits (~8000x improvement)

[ sensor id       ] [ type    ] [ value ]
0000 0000 0000 0001  0000 0001  0000 0001

It is now trivial, and extremely fast to update the sensor value on the receiving side. Scalar values might be encoded as 32 bit floats, while complex statuses will take up an arbitrary amount of data depending on the sensor.

The point is that sometimes, going “backwards” is actually the low hanging fruit, and sometimes there’s a lot of it to pluck.

The design of a system should take the environment into consideration as well. If you’re writing a module for a client where you have ample bandwidth, and time to market matters, it would be most appropriate to use wasteful, but fast tools. This means that as part of a development contract, the baseline environment must be described. Failure to do so will eventually lead to problems when the client comes back, angry that the system you wrote in 2 days doesn’t work via a dial-up connection.

 

CamStreamer

Axis does not support RTMP as a transmission protocol in their firmware (correct me if I am wrong about this), this seems to be an omission as RTMP offers a pretty good way for the camera to initiate an outbound connection to an RTMP service somewhere outside the network. I believe this is how DropCam works (again, an assumption).

As most probably know (or should know), an Axis camera is basically a small Linux computer with a camera attached. Axis will let you download a development environment, which you can then use to compile applications for the camera. This is called ACAP.

Before compiling an RTMP library for ACAP, I decided to check if someone else had done this. Turns out CamStreamer had done so, and I decided to give it a whirl.

Setup is extremely simple, and the UI looks modern and clean. It has support for some of the most popular streaming services, but I needed the RTMP ingest server (Universal)  as I have my own RTMP servers in the cloud.

Options

Setting this stuff up was trivial,

generic

The system pretty much worked like a charm with a single exception. During configuration, there was an error message telling me that the camera could not reach an outside host on port 1935 (the RTMP port). The error message is helpful and accurate. But later, I lost my internet connection completely (a screw-up at my ISP, where they switched me back to DHCP) and I was unable to enter the RTMP configuration page on the camera entirely. I guess this is because the HTML/configuration is served from a host in the cloud. I did not dig into it, but it struck me as a bit odd.

The ACAP application is pretty expensive: $299 for a 1-camera license (at the time of writing). This is pretty steep as you can hook up a Raspberry-PI type device that will do the same for several cameras for less than half that.

I am not sure why RTMP support is not built into more cameras (again, correct me if I am wrong). The “push” model of RTMP is almost perfect for cloud-based solutions, so why this is not standard fare for IP cameras is beyond me.

While Flash Video uses RTMP, RTMP can be used w/o Flash being involved at all. E.g. if I publish a stream via RTMP to EvoStream, I can view the stream via RTSP instead (and what can you do with RTSP streams and your favorite VMS?). It’s true that the reliance on TCP for RTMP introduces a bit of latency, but in practice, this is hardly a problem (if you are going to use a cloud solution, latencies are going to be pretty high compared to a local one).

Are You Diffident?

It always amused me when someone says “my personal opinion”. I find it strange, because the “personal” part is superflous. If the person says “my opinion is that red is a nice color”, I assume that the person means what he says: To him, red is a nice color.

If I then give him a red shirt, he says “don’t like it, I hate red”, I would assume some mental illness at play…

“but.. but.. you just said…”, I stutter

“yes, but that was not my personal opinion”

“WAT?”,

puzzled

Opinions don’t have to be personal, sometimes you’ll read “it is the opinion of the court” and things of that nature. But in those cases, it’s pretty clear that it is not the opinion of the person saying the words, that we are talking about. It would be exceedingly weird if the court clerk said “in my opinion, the accused is guilty”.

There are people with mental issues that have trouble with this concept. It is known as Dependant Personality Disorder. It basically means that you can’t have an opinion on anything, you constantly have to ask someone else what their opinion is and then act in accordance with that.

Someone who is deeply narcissistic (borderline?) might assume that everyone in the world, besides themselves, ought to suffer from DPD, and become upset and frustrated when people have opinions that do not align with what they are preaching.

The truth is that finding factual, verifiable information about IP cameras and software is getting easier every day (and this is an old video). Like most people, I don’t much care for what salespeople are saying if it can’t be verified or measured. If the salesperson can provide the raw data, I’ll take it. I will form my own opinion based on what I see. I don’t need some Gríma Wormtongue whispering into my ear.

grima2

With the commoditization of IP cameras, increasing demand for true interoperability we’re getting to a point where facts are valuable, whereas opinions are not (yep, this blog is free!!!). In some cases though. arguments and opinions may be based, not on unbiased interpretation of facts, but instead it is shaped by grudges and anger.

If you are paying for facts, you definitely should demand full disclosure, or if you’re not, you need to ask yourself, am I reading verifiable facts, or just bullshit? You might ask: Are manufacturers paying (directly, or indirectly) the one stating opinions about either the manufacturers products, or the products of the manufacturer’s competitors? If you’re being lied to in the full disclosure, you might be lied to elsewhere.