ISC West 2012

65 days since my last post. Well. We have been busy (we still are), but I need to spend a little time updating my blog every once in a while. ISC West is over, and I think now is a good a time as any.

We released Ocularis-X at the show. We think it is pretty cool, although I was more than a little worried that people would just look at it and go “yeah, it’s a web client – big deal”. Behind the covers we are doing some fancy footwork to offer the client high framerate, high resolution video over limited bandwidth. But to the end user it looks a lot like what everyone else is showing – except we demoed it over a tethered cellphone, and not on a LAN (at the booth we had to use wire because the wireless signals are just too unreliable).

What you can’t see is the flexibility that we’ve built into the server. Changing the protocol to suit Android devices took less than 5 minutes of coding, and I should also mention that the transcoder is a big bowl of dogfood too. Dogfood in the sense that the transcoder is using the Ocularis SDK components, so the web team were pretty much like a 3rd party developer, except they were in striking distance of yours truly, and could kick my behind when things were missing in the SDK.

Enough with the self-praise.

I spent a fair amount of time wandering aimlessly around the many booths at the show. Some booths were painfully vacant, and sometimes the presentations had nothing to do with the products at all. One company had a magician pull in the crowds. I wonder how many relevant leads that will yield. Everybody and their dog got scanned when they were watching the spectacle. Axis’s presentation was right across from ours (Brad is a freakin’ genius on stage), so there were pretty much non-stop presentations going on.

I love the Axis lightpath stuff. I think the lightpath idea is much, much more interesting than just high megapixel cameras. One company had a banner saying 1 camera replaces 96 VGA cameras. I’d take 96 VGA’s over one high megapixel camera any day. People keep educating me on this, but I probably will never learn. If that camera goes down, by the same logic, you are losing 96 (VGA) cameras. I am not against megapixel at all, I just don’t think it is a meaningful statement. Megapixel is part of a natural evolution – higher fidelity (although not all pixels are created equal), but one high quality vantage point can never replace multiple vantage points of lower quality. It’s apples and oranges. The 1 billion pixel camera reminded me of an old TED presentation of something called “Seadragon”, today it is called “deep zoom” I believe – pretty cool stuff.

The best thing about going to the show was to get to meet a lot of folks that use the software we wrote. Their feedback is a great source of inspiration to me – after a couple of beers, I usually get plenty of ideas, but talking for real users and dealers put things a little more in perspective.

It was good to finally meet you all in person. Hopefully we will meet again.

A Bug vs QA

Over on IPVM’s LinkedIn group there’s a discussion about compensation if a client encounters defective software or hardware. Vincent Tong brought up a point which was readily dismissed as a meaningless, semantic discussion – he pointed out that you need to discriminate between QA problems and bugs.

Here’s my defense of Vincent.

The term “bug” comes from the days when little critters – real bugs – would short electrical connections and cause machines to stop working. It’s a funny mental image, and so when poor soldering or routing on boards caused devices to malfunction, it was also called “bugs”. Naturally, programmers also wanted to blame someone else, so we decided to call mistakes “bugs” too.

Now, plenty of companies have experienced faltering, physical circuits. This week I “re-flowed” an old PS3. What happens is that the soldering between the chips and the motherboards come loose, to re-flow you melt the solder-balls ever so slightly to make them re-connect, the re-apply thermal grease and re-assemble. The issue is not entirely uncommon, but the question is – is it a bug?

To the end user, the semantic discussion is useless – the damn thing won’t work, but to manufacturers the distinction is extremely important. If you want to have a symbiotic relationship with your manufacturer, you should know, and appreciate, the difference too.

When people ask me if our software will work under XYZ conditions, I have to make an educated guess. First of all, the conditions are never described fully – did you specify the firmware of the camera, the brand and firmware of the switches, the cable lengths, the OS version, the language settings on the OS, the EFI version, the network adapter and so on. You don’t. And nor do we test every single permutation of hardware and software out there – it would simply be impossible to do.

The cell phone is broken!

I know that some of you are probably thinking – if they only used [insert favorite development model here] then it would be bug-free! My question is then – when was the last time you saw a 100% error free software that did something meaningful? Even my Ubuntu server needs updates every now and again – As soon as you offer advanced human interaction, it becomes real hard to avoid bugs.

This is why we need real good integrators, that educate the customer to pick tried and tested combinations of hardware, software and settings that we know to work well. But sometimes we encounter a combination of settings that just doesn’t work, and then we look like idiots.

I was told it could haul 2000 pounds!

This is completely different from a camera with a defective soldering or routing on the motherboard. It is completely different from a damaged CMOS sensor. When these things are discovered, the manufacturer should send you a replacement of the same device, and go whip the people assembling the damn thing. If you get too many of these duds and you don’t handle them well, the market will/should soon put them out of business. When you encounter a software bug, someone has to change the code, recompile and send you a new binary file.

This is the difference – the resolution to a bug is a different product, while a resolution to a QA problem is the same product (only this time, assembled correctly).

Alternatively, clients can and will routinely withhold payment if the system (in its totality) is not working well. If an integrator decides to make some obscure combination of hardware and software to make the sale without ever doing even basic testing before deployment, then theres a chance that things are not going to work. A great integrator will know what works and where to tread with care. Gunslingers will make the unreasonable expectation that every single combination must work, and demand compensation if it doesn’t – regardless of how exotic the permutation is.

You may also encounter contracts where the integrator is penalized if the system is not delivered to spec and on time, but that requires that you actually spec everything out in advance, and in 99.99% of those cases the manufacturer will be able to deliver (as you are able to test beforehand) and if they don’t they will find some detail in the contract that bails them out.

Analog vs IP Video

In the good old days, you had to have a guy walk in front of an automobile, ringing a bell to warn pedestrians that a car was coming (at walking pace), it was unreliable, difficult to operate, and one false move and it became an instant death trap. Horses, on the other hand, were easy to replace, they all fit in the old barn, and they all ate pretty much the same food.

The car analogy is not to say that IP is the car of today compared to a horse; the analogy is that when cars first came about, it took a long time before they became as homogeneous as they are today. Back then it was very difficult to see cars as a viable alternative to the horse. The point I was trying to make is that we are busy designing a better car, while others are convinced that Horse 2.0 is the way to go.

So is IP really better than Analog?

A DVR is analog – right?

Well… Let’s define the terms a little, to avoid the semantic confusion. In this discussion, “analog” refers to the transmission mechanism from the camera to the recording device. “analog” means that the camera sends an analog, uncompressed NTSC/PAL signal directly to the recording device. Whereas the an IP camera captures the image, compresses it and sends it via an IP network to the recording device.

The recording device may be labelled “DVR” or “NVR”, but in most cases the internal components are roughly the same. A DVR usually comes with a framegrabber card preinstalled that allows the recorder to capture the analog video and store it in digital form on a storage medium (is this starting to sound like lawyer-speak?). Likewise, an NVR may be retrofitted with a framegrabber card too, and thus the DVR and NVR becomes almost indistinguishable. Therefore, the discussion is not about DVR vs NVR, but rather analog vs. digital transmission of video.

Cabling
If you already have cabling in place, or if the placement of the cameras is such that you can’t cluster the cameras, then the cabling part of it is equal. But if you can do clustering, it is extremely cheap to do with IP cameras. Also, as Mark Schweitzer pointed out in the LinkedIn forum, IP comes with a built-in upstream channel, so if you ever need to replace a fixed camera with a PTZ, you do not need any additional cabling. Wireless transmission can also be achieved and you can monitor a camera (or a cluster of cameras) via the internet. As far as I can tell, HDcctv cannot piggyback off cat5, but need new HD-SDI cabling (I’m sure Todd Rockoff can clearify on the cabling requirements). I don’t know if SDTI is commonly used in HDcctv installations either.

Image Quality
Analog comes in a few flavors; the most common is NTSC/PAL(and their variations). NTSC has a resolution of 486 lines which in many cases is too low to identify faces unless they are very close to the camera. IP allows you to pick cameras that fit the purpose; cheap, low res cameras for overview, more expensive HD cameras for details and so on. If you so desire, you can easily replace one camera with another of higher or lower resolution. HDcctv seems to allow 1080P (2MP) as the maximum resolution. I think the increased flexibility of IP makes a winner overall.

Live viewing
For those who look at video, live, and respond to it, low latency and fluidity (high framerate) is important. High resolution less so. The reason is that you do not need crisp video to see that someone is fighting on the street, it is only when you go back to investigate and later go to court that the high resolution is real important (identification of license plates etc). Some IP cameras allow you to run 2 streams at the same time. One that gets recorded and one which is called up on demand when you want to view live video. Naturally, you are constrained by the bandwidth available to you as well as the ping-time. If you are cabling like you would an analog installation, you would have no problems with latency or quality at all.

On the other hand, if your IP camera does not provide dual streaming capabilities, then what you record is is what you get to see live. This means that if you run a low framerate and high compression then your live view will reflect that. On the other hand, you can always replace the camera with a better one that supports dual streams.

Playback
Even if the transmission of analog video is lossless the recorder will compress the video using the same compression technology as the IP camera. Any compression artifact introduced by the IP camera will also be introduced by the recorder as it compresses the video. However, while the IP solution may provide playback video in substantially higher resolution than the live video feed (which has an emphasis on framerate and latency), the analog solution can never provide higher quality than the live feed. What you see live is the highest “quality” that you will ever see.

Again, apples to apples, if IP is deployed using analog cabling conventions, you can certainly get the same live/playback quality as an analog system (e.g. VGA/30fps). Furthermore, your recorded video may be of much better fidelity than your live video (e.g. 1080p/10fps). This makes it possible to identify faces and license plates on the recorded video as you are conducting an investigation. It must be stressed that this is not always possible to achieve, bandwidth constraints are always a factor that must be taken into consideration. You should never expect to get 5MP, high quality, 30 fps recordings over a dial-up internet connection.

Scalability
IP cameras is like a BYOB party. Everyone brings their own beer, so you just need to consider how many people will physically fit in your pad. IP cameras already do the compression for you, so the recorder simply needs to pipe the data to the hard-drives. In an analog system, the recording device is handing out the beer as people arrive. The host might have enough beer for a 16 people, but once you are out of beer the party stops. Furthermore, some IP cameras will allow you to do motion detection, or even video analytics directly on the camera (this technology is still in its infancy though). Obviously, a system based on analog transmission can also scale, by adding more/stronger/faster recorders. In terms of scalability I think IP eeks out the advantage, but not by much.

Conclusion
I think the prudent investment is not in more barns for horses or investing heavily in horse 2.0 (which requires special pavement to achieve the speed of a car).

With that said – Happy Holidays folks

Your Sourcecode is Worthless

When Google decided to do Android, they didn’t go and copy Apples iOS sourcecode. They didn’t have to – Google have enough great engineers that they could do their own implementation of iOS’s features; the value of Apple/iOS is in the ideas and the execution.

While a lot of people have ideas, they usually lack the technical ability to bring them to fruition, and even if they do, they might miss the elegance and finesse of a truly awesome solution. Copying source code means you are already behind the curve, you are not gaining anything, instead you are teaching your team that plagiarism is of higher value than innovation. Followship instead of Leadership.

Teach your team to be innovative, to execute ideas well, and you can publish your source-code on a public forum if you want*

*Never publish your source-code, but THINK of it as published, that will make you run faster and become less complacent.

NVR Integration and PSIM

Axis makes cameras, and now they make access control systems and an NVR too. Should the traditional NVR vendors be concerned about it?

Clearly, the Axis NVR team is better positioned to fully exploit all the features of the cameras. Furthermore, the Axis only NVR does not have to spend (waste?) time, trying to fit all sorts of shapes into a round hole. They ONLY have to deal with their own hardware platforms. This is similar to Apple’s OS X that runs so smoothly because the number of hardware permutations are fairly limited.

What if Sony did the same? Come to think of it, Sony already has an NVR. But, it’s no stretch of the imagination to realize that a Sony NVR would support Sony cameras better than the “our NVR does everything”-type.

In fact, when you really look at the problem, an NVR is a proprietary database, and some protocol conversion. To move a PTZ camera, you need to issue various different commands depending on the camera, but the end result is always the same: the camera moves up. Writing a good database engine is really, really hard, but once you have a good one, it is hard to improve. The client and administration tools continue to evolve, and become increasingly complex.

Once it becomes trivial to do the conversion, then any bedroom developer will be able to create a small NVR. Probably very limited functionality, but very cheap.

The cheap NVR might have a simple interface, but what if you could get an advanced interface on top of that cheap NVR? What if you could mix cheap NVRs with NVRs provided by the camera manufacturers, and then throw in access control to the mix? You get the picture.

If you are an NVR vendor, it is going to be an uphill battle to support “foreign” NVRs. If Milestone decided to support Genetec, it would take 5 minutes for Genetec to break that compatibility and have Milestone scramble to update their driver. Furthermore, the end user would have to pay for two licenses, and the experience would probably be terrible.

The next time an NVR vendor says “we are an open architecture”, then take a look at their docs. If the docs do not describe interoperability with a foreign client, then they are not open. An ActiveX control does NOT equate “open”. Genetec could easily support the Milestone recorders too, but it would be cheaper and easier for Genetec to simply replace existing Milestone recorders for free (like a cuckoo chick).

In this market, you cannot get a monopoly and charge a premium. The “need to have” threshold is pretty low, and if you charge too much, someone will make a new, sufficient system and underbid you. Ultimately, NVRs might come bundled with the camera for free.

So, what about PSIM? Well, we started with DVR (which is really a good enough description of what it is), but then we decided to call it an NVR to distance ourselves from the – clearly inferior DVR. Then we weren’t satisfied with that, so then it became VMS. Sure, we added some bells and whistles along the way (we had mapping and videowall in NetSwitcher eons ago), so now we call it PSIM. It does the same as the old DVR. I think this kind of thing is called “marketing”.

Is ONVIF a complete failure?

I like to sit on the sideline and critique the work that other people did. I tend not to participate in committees as I find them totally counterproductive.

To make a protocol a success, I think it should be simple. Now, simplicity is in the eye of the beholder, but take the original HTTP proposal. One page.

ONVIF is not just one page. It is a LOT of pages, a few concepts but a LOT of room to interpret things differently and mess things up (don’t forget to add developers tendency to make funny errors to the mix).

I am told that ONVIF is already creating problems in the real world. An NVR vendor recently announced that a new “Samsung ONVIF driver” was available. The idea that you write ONVIF drivers for particular vendors tells me that while ONVIF might not be broken per se, there are certainly problems pretty much across the board (please correct me if I am wrong).

ONVIF is really a great example of two problematic concepts. Design by committee and the second system effect. While the committee part is fairly obvious, the second system effect might need some clarification: Most people that participated in the committee had already had experience with simple HTTP based interfaces. Now, instead of picking the BEST parts, keeping it simple and then incrementally add functionality, they decided to make this the “mother of all rewrites”. No more simplistic HTTP, no, let’s go all out and throw the technology du jour at it. What about SOAP? Hell YEAH!! I guess the ONVIF guys could have chosen CORBA as well, because when CORBA was hot the community wanted to use CORBA for everything too.

Here’s one example of how the ONVIF committee, in my opinion, have compromised the user for the sake of the technology: Events are never pushed from the camera to the NVR, instead the NVR has to POLL for events. This may or may not be a problem, if you want your system to respond fast you need to poll more often. I think it is a poor design. I don’t need my app to act like a kid in the back seat asking me “are we there yet” every 30 seconds (or every second if you want something that seems a little closer to real time).

Now, I don’t pretend to know the motivation for this weird design and I suppose that there is some merit to it (I just don’t know what it would be). An alternative could have been a HTTP POST with some XPath and an expiration, the response would then be a multipart/mixed response, each payload corresponding to an event.

Don’t take this the wrong way, the ONVIF spec is a magnificent specimen of work, but I believe you have over-engineered the solution. All we needed was for you guys to agree on the same freaking URI to get video and we’d be done, instead we got something terribly complex that I believe will be too difficult for a lot of people to implement.

Flamebaiters

Not to be mistaken for the common internet troll, the flamebaiter will post articles and comments that are unnecessarily provocative or offensive in order to attract traffic.

Many forums have died over the years as they started to self-radicalize. As the forum attracts people who think alike, and start to repel dissenting voices, the participants get increasingly convinced that the forums ideas represent the true order of things.

Every active participant, in almost any blog or forum, has an agenda. The most honest thing, in my opinion, is to state your agenda clearly. That puts your posts in the correct context.