Here’s my defense of Vincent.

The term “bug” comes from the days when little critters – real bugs – would short electrical connections and cause machines to stop working. It’s a funny mental image, and so when poor soldering or routing on boards caused devices to malfunction, it was also called “bugs”. Naturally, programmers also wanted to blame someone else, so we decided to call mistakes “bugs” too.

Now, plenty of companies have experienced faltering, physical circuits. This week I “re-flowed” an old PS3. What happens is that the soldering between the chips and the motherboards come loose, to re-flow you melt the solder-balls ever so slightly to make them re-connect, the re-apply thermal grease and re-assemble. The issue is not entirely uncommon, but the question is – is it a bug?

To the end user, the semantic discussion is useless – the damn thing won’t work, but to manufacturers the distinction is extremely important. If you want to have a symbiotic relationship with your manufacturer, you should know, and appreciate, the difference too.

When people ask me if our software will work under XYZ conditions, I have to make an educated guess. First of all, the conditions are never described fully – did you specify the firmware of the camera, the brand and firmware of the switches, the cable lengths, the OS version, the language settings on the OS, the EFI version, the network adapter and so on. You don’t. And nor do we test every single permutation of hardware and software out there – it would simply be impossible to do.

The cell phone is broken!

I know that some of you are probably thinking – if they only used [insert favorite development model here] then it would be bug-free! My question is then – when was the last time you saw a 100% error free software that did something meaningful? Even my Ubuntu server needs updates every now and again – As soon as you offer advanced human interaction, it becomes real hard to avoid bugs.

This is why we need real good integrators, that educate the customer to pick tried and tested combinations of hardware, software and settings that we know to work well. But sometimes we encounter a combination of settings that just doesn’t work, and then we look like idiots.

I was told it could haul 2000 pounds!

This is completely different from a camera with a defective soldering or routing on the motherboard. It is completely different from a damaged CMOS sensor. When these things are discovered, the manufacturer should send you a replacement of the same device, and go whip the people assembling the damn thing. If you get too many of these duds and you don’t handle them well, the market will/should soon put them out of business. When you encounter a software bug, someone has to change the code, recompile and send you a new binary file.

This is the difference – the resolution to a bug is a different product, while a resolution to a QA problem is the same product (only this time, assembled correctly).

Alternatively, clients can and will routinely withhold payment if the system (in its totality) is not working well. If an integrator decides to make some obscure combination of hardware and software to make the sale without ever doing even basic testing before deployment, then theres a chance that things are not going to work. A great integrator will know what works and where to tread with care. Gunslingers will make the unreasonable expectation that every single combination must work, and demand compensation if it doesn’t – regardless of how exotic the permutation is.

You may also encounter contracts where the integrator is penalized if the system is not delivered to spec and on time, but that requires that you actually spec everything out in advance, and in 99.99% of those cases the manufacturer will be able to deliver (as you are able to test beforehand) and if they don’t they will find some detail in the contract that bails them out.

I am now starting to question if that was a wise decision.

When clients ask for a 100 camera view, I used to wonder why. No-one can look at 100 cameras at the same time. But then I realized that they don’t actually look at them the way I thought. They use the 100 camera view a “selection panel”. They scan across the video and if they see something out of place they maximize that feed.

I am guessing here, but I suspect that in live view, you want to see “a little bit of everything” to get a good sense of the general state of things. When something happens, you need a more focused view – suddenly you go from 100 cameras to 4 or 9 cameras in a certain area.

Finally, when you go back and do forensic work, the use pattern is completely different. You might be looking at one or two cameras at a time, zooming in and navigate to neighboring cameras quite often.

Hmmm… I think we can improve this area.

I guess they imagined that screen resolutions would just “grow and grow” and we’d all have screens with phenomenal resolution in the future. But then what happened? Every other panel out there is 1080p for workstations (save for a few exceptions), while most laptops come with 1400×800 or something in that range. Smaller panels as found in smartphones have a high pixel density but small displays, and, I might add Microsoft hardly has any presence in that segment.

Man, I am fed up with WPF right about now.

The car analogy is not to say that IP is the car of today compared to a horse; the analogy is that when cars first came about, it took a long time before they became as homogeneous as they are today. Back then it was very difficult to see cars as a viable alternative to the horse. The point I was trying to make is that we are busy designing a better car, while others are convinced that Horse 2.0 is the way to go.

So is IP really better than Analog?

A DVR is analog – right?

Well… Let’s define the terms a little, to avoid the semantic confusion. In this discussion, “analog” refers to the transmission mechanism from the camera to the recording device. “analog” means that the camera sends an analog, uncompressed NTSC/PAL signal directly to the recording device. Whereas the an IP camera captures the image, compresses it and sends it via an IP network to the recording device.

The recording device may be labelled “DVR” or “NVR”, but in most cases the internal components are roughly the same. A DVR usually comes with a framegrabber card preinstalled that allows the recorder to capture the analog video and store it in digital form on a storage medium (is this starting to sound like lawyer-speak?). Likewise, an NVR may be retrofitted with a framegrabber card too, and thus the DVR and NVR becomes almost indistinguishable. Therefore, the discussion is not about DVR vs NVR, but rather analog vs. digital transmission of video.

Cabling
If you already have cabling in place, or if the placement of the cameras is such that you can’t cluster the cameras, then the cabling part of it is equal. But if you can do clustering, it is extremely cheap to do with IP cameras. Also, as Mark Schweitzer pointed out in the LinkedIn forum, IP comes with a built-in upstream channel, so if you ever need to replace a fixed camera with a PTZ, you do not need any additional cabling. Wireless transmission can also be achieved and you can monitor a camera (or a cluster of cameras) via the internet. As far as I can tell, HDcctv cannot piggyback off cat5, but need new HD-SDI cabling (I’m sure Todd Rockoff can clearify on the cabling requirements). I don’t know if SDTI is commonly used in HDcctv installations either.

Image Quality
Analog comes in a few flavors; the most common is NTSC/PAL(and their variations). NTSC has a resolution of 486 lines which in many cases is too low to identify faces unless they are very close to the camera. IP allows you to pick cameras that fit the purpose; cheap, low res cameras for overview, more expensive HD cameras for details and so on. If you so desire, you can easily replace one camera with another of higher or lower resolution. HDcctv seems to allow 1080P (2MP) as the maximum resolution. I think the increased flexibility of IP makes a winner overall.

Live viewing
For those who look at video, live, and respond to it, low latency and fluidity (high framerate) is important. High resolution less so. The reason is that you do not need crisp video to see that someone is fighting on the street, it is only when you go back to investigate and later go to court that the high resolution is real important (identification of license plates etc). Some IP cameras allow you to run 2 streams at the same time. One that gets recorded and one which is called up on demand when you want to view live video. Naturally, you are constrained by the bandwidth available to you as well as the ping-time. If you are cabling like you would an analog installation, you would have no problems with latency or quality at all.

On the other hand, if your IP camera does not provide dual streaming capabilities, then what you record is is what you get to see live. This means that if you run a low framerate and high compression then your live view will reflect that. On the other hand, you can always replace the camera with a better one that supports dual streams.

Playback
Even if the transmission of analog video is lossless the recorder will compress the video using the same compression technology as the IP camera. Any compression artifact introduced by the IP camera will also be introduced by the recorder as it compresses the video. However, while the IP solution may provide playback video in substantially higher resolution than the live video feed (which has an emphasis on framerate and latency), the analog solution can never provide higher quality than the live feed. What you see live is the highest “quality” that you will ever see.

Again, apples to apples, if IP is deployed using analog cabling conventions, you can certainly get the same live/playback quality as an analog system (e.g. VGA/30fps). Furthermore, your recorded video may be of much better fidelity than your live video (e.g. 1080p/10fps). This makes it possible to identify faces and license plates on the recorded video as you are conducting an investigation. It must be stressed that this is not always possible to achieve, bandwidth constraints are always a factor that must be taken into consideration. You should never expect to get 5MP, high quality, 30 fps recordings over a dial-up internet connection.

Scalability
IP cameras is like a BYOB party. Everyone brings their own beer, so you just need to consider how many people will physically fit in your pad. IP cameras already do the compression for you, so the recorder simply needs to pipe the data to the hard-drives. In an analog system, the recording device is handing out the beer as people arrive. The host might have enough beer for a 16 people, but once you are out of beer the party stops. Furthermore, some IP cameras will allow you to do motion detection, or even video analytics directly on the camera (this technology is still in its infancy though). Obviously, a system based on analog transmission can also scale, by adding more/stronger/faster recorders. In terms of scalability I think IP eeks out the advantage, but not by much.

Conclusion
I think the prudent investment is not in more barns for horses or investing heavily in horse 2.0 (which requires special pavement to achieve the speed of a car).

With that said – Happy Holidays folks

While a lot of people have ideas, they usually lack the technical ability to bring them to fruition, and even if they do, they might miss the elegance and finesse of a truly awesome solution. Copying source code means you are already behind the curve, you are not gaining anything, instead you are teaching your team that plagiarism is of higher value than innovation. Followship instead of Leadership.

Teach your team to be innovative, to execute ideas well, and you can publish your source-code on a public forum if you want*

*Never publish your source-code, but THINK of it as published, that will make you run faster and become less complacent.

<Path Stroke=”Black” StrokeThickness=”1″  Data=”M 0,10 A 10,10 90 0 1 10,0 H 90 A 10,10 -90 0 1 100,10 V 60 A 10,10 -90 0 1 90,70 H 80 L 70,80 L 60,70 H 10 A 10,10 -90 0 1 0,60 V 10″ Fill=”White” />

…But that’s like judging a fish by it’s ability to climb trees (as Einstein supposedly said once).

Let’s flip it around, and ask how well an analog PTZ camera can look in two directions at once. Let’s ask if analog PTZ can do a tour at 90 degrees per second, 24-7-365 without breaking. Let’s try and do analog PTZ in Houston, from an office in New York on a shoestring. Let’s ask if we can change our minds and zoom in on a different area after the video was recorded.

Now, if an IP based optomechanical PTZ camera was given the same budget as the cost of wiring an analog one, then you would not be able to tell them apart at all. And I am guessing you don’t really need to spend the full budget to get equal performance – you can probably get good (perhaps not as good) performance a lot cheaper.

But what if you really wanted to replace a mechanical PTZ with a MP camera?

I guess a lot of installation were getting mechanical PTZ’s in the past because there was no other choice. Now there is. It’s fairly cheap to install 3 fixed cameras vs. 1 PTZ (simply because you don’t need to pull 3 cables all the way back to the recorder – you can pop in a POE switch and cluster the 3 cameras). If you then put in 3 decent cameras you are golden. You even get to see things from 3 vantage points – something PTZ will never do. Even if a vandal breaks one, you still have 2 others that are recording.

It is true that right now, the cost of 2 additional camera licenses are a burden, but I think that cost will come down dramatically over the next 24 months.

But a mechanical PTZ camera is really equivalent to a Gigapixel camera. If you do the maximum zoom level, and do a full pan-tilt of the area you get a huge resolution. If you were monitoring highways it would make sense to have an optical PTZ at the intersections which would allow you to zoom in much more than the MP would ever let you.

Another disadvantage to HD cameras are that they take up a lot of space and they require a lot of processing power to decode. This is mostly an issue for the client side developer (as we need to decode the frames to show them to you ), but an issue nonetheless. Some people will just compress the crap out of those feeds, but that totally negates the purpose. You might as well use a lower resolution camera then. Sometimes the framerate gets lowered to the point where you might as well be looking at a slideshow – but that might just be good enough for the user.

So I think Todd Rockoff is correct. HD and PTZ are complimentary.

One problem that I am aware of is if you need to pre-allocate space for each camera. A camera with very little motion might record for 100 days in a 100 GB allocation, while a busy one might have just 1 day. Change the parameters and it gets real hard to figure out what a reasonable size should be.

But say that you pre-format the total storage you need for the entire system, and then let all the cameras share the storage on a FIFO basis. This way, all cameras would have roughly the same amount of time recorded in the database.

My, decidedly unscientific tests, show that writing a large block of data to a continuous area on the disk is much faster than writing to a file that is scattered across the platters. Disk drives now have large caches and command queuing, but these mechanisms were designed for desktop use, and not a torrent of video data being written and deleted over and over again.

Some people balk at the idea that you pre-format the disk for reasons I simply do not understand. If you have a 100 TB storage system, I would expect that you’d want to use the full capacity of the disk. There are no points awarded for having 20% of the disk empty, so why do people feel that pre-allocation is bad?

Any takers?