Hyperdistributed IP video

Verkada and Meraki is freaking everyone out. They’re selling direct, and they don’t respect the hierarchy. They’re seemingly attracting big $$$ from VC’s, and they’re landing million dollar deals at a fairly impressive clip.

These spry Silicon Valley kids with their typescript, Kubernetes, hoodies and sandals don’t give a damn about the gospel we’ve long established as the only truth under the sun.

The jury is still out on their approach. Right now, to me, it smells a lot like snake-oil. There’s a distinct aura of petroleum coming from the flask the energetic salesman is offering. I’ll let someone else drink the tonic and see how it goes before I’m willing to ingest this swill. But, make no mistake – they’re on to something, and they are probably right to taunt the rest of the industry.

So, why is the hyperdistributed model so good?

Let’s say you have a campus and need 500 cameras. With a traditional solution, those 500 cameras would be streaming – non-stop – to a central location (a server room usually). At the server room, you’d have a very expensive storage solution, typically with some sort of redundancy. Along with the storage, you’d need a few PC’s who’s main objective is to simply read from the camera, and store the feed for later review. A review that 99% of the time never happens. Video is compressed, sent across the network, stored on a disk, and then deleted when it gets too old. If the data-center goes down, it takes with it ALL recordings, and you’re basically blind. You can then install redundant data-centers (certainly not free!), which brings you maybe a little close to a distributed model.

It might be that you have an area that doesn’t have great WiFi network coverage, but you might have cellular access. Do you then establish a mesh-network all the way to the shed? Or do you let the camera stream 24/7 across the LTE network? Both are costly, and the cost is sunk if you later decide to move the camera.

I see two major problems with the hyperdistributed “Veraki” solutions: 1) if the camera is damaged, the evidence is gone, and 2) you’re currently locked in with either if you go that route.

The first problem is simple enough. Veraki simply offer a box that physically separates the imager from the storage via a cable, such that the recording can be better protected and will not disappear with the destruction of the camera.

The other is harder, because establishing a standard will almost immediately cause commoditization and ultimately loss of margins. I don’t think that’s a solvable problem, and the hypothetical risk is that someone actually establishes a standard for this sort of thing. Knowing the industry, that’s not going to happen in the next 10 years.

I don’t see the lack of camera setting, sync playback and so on as a long term problem. Currently, the idea is that all things should be web-based, but they’ll soon learn that the web was not made for video surveillance. If they’ve got a good database design, it shouldn’t be a problem to write a native app that could offer a more feature-rich experience for those who crave it. Spotify, for example, requires an installed app on the PC to play, the web site only offers account management.

If memory serves, Milestone actually worked on putting a lightweight version of their recorder inside cameras, and I think they signed some partners, but did it ever go anywhere? Such a solution would be strongly preferred, as “anyone” can communicate with a Milestone VMS and even write a client from scratch based on it.

Some companies will respond to this new threat by digging themselves deeper into a hole, while claiming to be reaching for the stars.

The (Real) Problem With Cybersecurity

Having been in the sausage-factory for a long time, I’d like to share some thought about what I think is a problem when it comes to cyber-security.

Contrary to popular belief programmers are humans; we make stupid mistakes, some are big, some are small. Some days we are lazy, others we are energetic and highly motivated. So (exploitable) bugs inevitably creep in, this is just a fact of life.

The first step in writing robust and secure code is in the design and architecture. The next step is to have developers with good habits and skills, and finally you run a good selection of automated tests on the modules that make up the product.

But consider that a VMS typically runs on an OS that the vendor has very little control over, or uses a database (SQL server, mySQL, MaxDB etc) that is also outside the manufacturers reach. Furthermore, the VMS itself uses libraries from 3rd parties, and with good reason too. Open Source libraries are often much better tested and under an extreme amount of scrutiny compared to a VMS doing a homemade concoction reviewed by just a few peers, but they too have bugs (just fewer than the homebrew stuff usually).

Inevitably, someone finds a way to break the system, and when it happens, it’s a binary event. The product is now insecure. You can argue all you want that the other windows and doors are super-secure, but if the back door is open – who cares about the lock on the window?

To be fair, if the rest of the building is locked down well, then fixing the broken door may be a smaller event.

Contrast to a system that is insecure by design. Where fixing the security issues requires changes to the architecture. We’re no longer talking about replacing a broken lock, but upheaval of the entire foundation. An end-user doesn’t know if the cracks are due to a fundamental issue, or something that just needs a bit of plaster and paint.

And this brings me to the real issue.

Say a developer politely asks demand that resources are allocated to fixing these issues, what do you imagine will happen? In some companies, I assume that a task-force is assembled to estimate the severity of the issue, resources are then allocated to fix the issue. A statement is issued so that people know to apply the patch (they’re not going to do it, but it’s the right thing to do). This is what a healthy company ought to do. A sick company would make the following statement: “no-one has complained about this issue, and – actually – we have to make money”.

A good way to make yourself unpopular (as a programmer) is to respond by saying that if the issue IS discovered, you can forget about making any money. Your market will be limited to installations who really don’t care about security. The local Jiffy-Lube who replaced their VHS based recorder with a DVR that just sits on a dusty shelf may truly not care. The system is not exposed in any way – it is a CCTV (Closed, being the operative word here). They’re fine. And the root password is written on a post-it note stuck on the monitor. But what about a power plant? What about a bank? an airport?

You might imagine that an honest coder with integrity would resign on the spot, but this doesn’t solve the problem. Employees are often gagged by NDAs and non-disparagement clauses, and while disclosure of security flaws is clearly protected by the first amendment, it is generally a bad idea to talk about these things. The company may suffer heavy losses and you are putting (unsuspecting) customers at risk by making these things public. The threat of legal action and the asymmetry (a single person vs a corporation) ensures that flaws rarely surface.

It’s also conceivable that the dumbass programmer, is wrong about the risk of a bug/design issue. A developer may think that a trivial bypass of privilege checks is “dangerous”, but customers might genuinely not care.

Who knows? During the Black Hat convention in 2013, IP cameras from different manufacturers were shown to be hopelessly unsafe. Didn’t seem to make any difference.

I referenced this talk in an earlier post as well.

4 years later, cybersecurity is all the rage, and perhaps people do care – but from what I can tell, it’s just a few SJWs who crave the spotlight who pretend to care. Whether the crazy accusations have merit is irrelevant, all that matters is that viewers tune in, and the show will get increasingly grotesque to keep people entertained. And if the freaks-show is not bringing in the crowds, you can always turn it into a sort of “anonymous facebook” where people can back-stab each other – like the bitchiest teenage girls used to treat each other.

What the industry probably needs to do, is to pay professional penetration testers to go to work on the systems out there. I’m not talking about the kind of shitty automated tests that are being done today. They are far, far from being sufficient. You need people like Craig Heffner in the video to go to town to get to the bottom.

Happy hacking.

Managing the Manager-less Process

Fred George has quite a resume – he’s been in the software industry since the 70’s and is still active. His 2017 talk @ the GOTO conference is pure gold.

His breakdown of the role of the Business Analyst at 19:20 is spot on. The role of the manager is even saltier (23:12) – “I am the God here”.

Well worth an hour of your life (mostly for coders).

As a side node there are two characters in the Harry Potter movies called “Fred and George”, making searches for “Fred George” a pain.

Paranoia?

Some time ago, Bloomberg ran an article claiming that Chinese computer components (in this case a motherboard) would be intercepted en route to customers and be modified to host a small chip that would allow the (evil) Chinese government to spy on the righteous.

It was an unusually sensational piece for Bloomberg, complete with a fake animation zooming in on a cartoon-styled motherboard, suggesting that Bloomberg knew, as a matter of fact, where the alleged chip was placed. They even showed the chip placed on top of a finger. I’d call it deceptive, because Bloomberg demonstrably did not have any physical evidence of the chip, so the motherboard zoom-in and finger-chip were fabrications. If I discovered a “rogue” chip on any of my devices, I can assure you, I would keep the evidence around. What person discovers a rogue chip on a motherboard, and then just discard it?

Because it’s very difficult, and often impossible to prove a negative, the burden of proof is on the accuser. It’s too easy to say that people roaming the certain internet forum is actually a front for exchange of immoral and perverse videos. The admin of the forum and its members would deny the allegations, and I’d just say – “of course they are denying it, it would destroy their business and reputation if they didn’t deny it”, and I would then demand that they prove they never exchanged sick videos. Can’t be done.

It all brings memories of Stephen Glass.

Does that mean that it is inconceivable that hardware from China is bugged? No, nor does it mean that evidence will never surface. All it means is that if you’re buying into the Bloomberg story, then you’re probably part of the problem.

It’s a problem when people start believe gossip simply because it supports their belief. Don’t like/can’t compete with the Chinese, then you’re likely to believe some gossip about “spy chips” that no-one so far has been able to prove existed.

At the same time, when there are vulnerabilities in chipsets from Intel, then that’s just an honest mistake.

I don’t trust anything, and you shouldn’t either. Instead, you should spend less time obsessing over gossip (as entertaining as it might be), and instead educate yourself on how to protect yourself from eavesdropping. I’m not suggesting you’ll ever get 100% security when dealing with computers – and I don’t care who the manufacturer is. Things are put together by humans, and we make mistakes (or perhaps we have a fallout with former allies who then promptly leaks our secrets), so it’s on you to take precautions.

Stay safe, and don’t spread rumors and gossip. Reserve judgment until you see the evidence, not before.

 

 

GDPR

IANAL

When EU makes laws (or any government entity, really), it’s a trial and error process. Traffic laws are simple. If you’re allowed to drive at 50 km/h, then driving 49 km/h is OK, driving 51 km/h is not. Social laws are much more complex. Tax law has become a massive pile of spaghetti filled with bugs that allow some entities to pay very little, and other entities to pay a lot.

The General Data Protection Regulation is an attempt to protect the privacy of citizens of the EU. Companies are not allowed to covertly collect massive amounts of information about people; they must inform the user, and they must delete personal information if asked to do so.

From the EU’s FAQ on the topic, you can find this passage

What constitutes personal data?
Any information related to a natural person or ‘Data Subject’, that can be used to directly or indirectly identify the person. It can be anything from a name, a photo, an email address, bank details, posts on social networking websites, medical information, or a computer IP address.

Added emphasis.

It seems to me that EU is thinking about companies like Facebook, Google and just about every other company that collects massive amounts of data about their users.

Most video surveillance systems are not capable of tracking people around the store, let alone recognize the same user as they visit the store on different days. Naturally, if you have this wonderful technology deployed, and you are actually identifying people in the feed, I believe that you might have to take a much closer look at GDPR and start posting notes and ask for consent (in clear language).

As I understand it, if regular video surveillance is in effect, you need to post information about this (you already are), and probably be able to document your retention policies if anyone asks.

Granted, if you start building cases about people, where you are collecting clips and images, then you def. need consent.

Adding a bit of Gaussian blur to the faces of people (when displayed in the client) is simply irrelevant in relation to GDPR. If you are storing the information, you have to inform and get consent.

I harbor no illusions that slick salespeople won’t try to use these new laws as a way to sell more snake-oil. I am also pretty certain that you’ll hear unverified anecdotes about companies that were fined because they did not blur the faces in the viewer application.

So, post a sign that says “video is recorded and kept for 30 days” and you’re good. But you have to actually abide by it. If you perform analytics (poor you), track people, correlate purchases and arrival times to POS readouts, then you have to post that too.

In Denmark, they’ve already announced that because this is such a complicated issue (mostly due to charlatans spreading FUD), they will be very lenient with non-compliance unless it is an intentional violation of the rules (e.g. a standard VMS running a 2 week loop in a store is not going to be fined for not posting a consent form at the door).

 

 

My Bitcoin Problem

I didn’t get enough of them…. ?

Back in the good old days, Hikvision NVRs part of an exploit that was used to mine Bitcoin, naturally, that was back when Bitcoin was used primarily to buy heroin and weapons via the darknet. Today, though, everyone and their dog is buying bitcoin like it was pets.com shares ca 2001,  and the hardware needed to mine coins today is a million times more powerful than a cheapo NVR.

First things first; why do we need “currency”. I think it’s worth revisiting the purpose, before moving on. Basically, “currency” is a promise, that someone (anyone) will “return the favor” down the line. In other words, I mow your lawn, and you give me an IOU, which I trade for some eggs at with the local farmer. The farmer then trades the IOU for getting picket fence painted by you (you then tear up the IOU).

Instead of crude IOU’s, we convert the work done into units of currency, which we then exchange. Mowing a lawn may be worth 10 units while doing the dishes is worth 5. In the sweet old days, the US had many different currencies, pretty much one per state. They served the same purpose. To allow someone to trade a cow for some pigs and eggs, some labor for food, food for labor and so on.

But pray tell, what politician, and what banker would not love to be able to issue IOUs in return for favors, without actually ever returning them?

Since politicians and bankers run the show, naturally, the concept got corrupted. Politicians and banks started issuing IOUs left and right, which basically defrauded you of your work. When you mowed the lawn on Monday, you would expect that you could exchange the IOU for a lawn mowing on Friday, but with politicians producing mountains of IOUs, you suddenly find that the sweat off your brow on Monday only paid for half the work on Friday.

This is classic inflation.

By the same token, it would be one hell of an annoyance if you mow my lawn on Monday, and now, to repay you, I would have to not only mow your damn lawn, but also paint your fence on Friday.

This is classic deflation.

What you want is a stable, and fair currency. That work you do on Monday can be exchanged for an equal amount of work on Friday.

You can then wrap layers of complexity around it, but at its core, the idea is that money is a store of work, and that store should be stable.  The idea that we “need 2% inflation” is utter nonsense. In a democracy, the government can introduce a tax on cash equivalent holdings if the voters so desire. This would be more manageable and precise than senile old farts in central banks trying to “manage inflation” by purchasing bonds and stock, with the predictable side effect that it props up sick and useless companies. The idea that you can get work done by just shuffling some papers around is an abomination in my book.

Bitcoin is an attempt at creating a currency that can’t be manipulated by (presumably corrupt or incompetent) politicians and bankers, but I think they’ve gone far, far away from that idea.

The people who are engaging in bitcoin speculation are not doing it because they want a fair and stable store of work (having discarded traditional fiat currency as being unstable and subject to manipulation). Instead, they do it, because, in the speculative frenzy, bitcoin is highly deflationary. You can get a thousand lawns mowed on Friday for the lawn you mowed on Monday. As a “stable currency”, Bitcoin has utterly failed. And we’re not even discussing the transaction issues (200K back-logged transactions, and a max of 2000 transactions every 10 minutes).

This happens because bitcoin is not a currency at all. It’s a simply the object underpinning a speculative bubble. And as it happens with all bubbles, there are people who will say “you don’t understand why this is brilliant, you see… ” and then a stream of illogical half-truths and speculation follows. People share stories about how they paid $100 for a cup of coffee 12 months ago when they used bitcoin to pay for it. But a cup of coffee in dollars cost about the same as it did 12 months ago, so while the dollar is being devalued by very mild inflation, and thus a much more stable store of work, bitcoin is promising free lunches for everyone.

People, for the most part, take part in this orgy with the expectation that at some point, they will settle the score for real currency – real dollars. Very few (and I happen to know one) will keep them “forever” on principle alone.

Furthermore, I don’t see any reason why the Bitcoin administrators wouldn’t just increase the self-imposed 21 million coin limit to 210 million of 2.1 billion coins. They already decided to create a new version, called Bitcoin Cash that essentially doubled the amount of bitcoin. That and the 1300 other cryptocurrencies out there makes it hard for me to buy into the idea that there is a “finite number of coins”. Not only that, to increase transaction speed to something useful, they are going to abandon the blockchain security, opening up for all sorts of manipulation (not unlike naked short selling of stock etc.)

And let’s not forget that before Nixon, the civilized world agreed to peg currencies to gold (a universal currency that could not be forged). In 1973, Nixon removed the peg from the US dollar and since then the number of dollars has exploded, and the value has dropped dramatically. In other words, what was a sure thing pre-1973, was suddenly not a sure thing.

This is not investing advice. You might buy bitcoin (or other crypto-“currencies”) today, and make 100% over the next few weeks. You might also lose it all. I would not be surprised by either.

 

Net Neutrality

You can’t be against net neutrality, and, at the same time, understand how the Internet works.

There is no additional cost to the IPS to offer access to obscure sites; it’s not like a cable package where the cable provider pays a fee to carry some niche channel that no-one watches.

Basically, net neutrality means that the ISP has to keep the queues fair; there are no VIP lanes on the Internet. Everyone gets in the same line, and are processed on a first come, first served basis. This is fundamentally fair. The business class traveler may be angered by the inability to buy his way to the front of the line (at the expense of everyone else), but that’s just tough titties.

It’s clear that not everyone has the same speed on the Internet; I live in an area where the owners association decided against having fiber installed, so I have a shitty (but sufficient) 20/2Mbit ADSL connection. My friend across the bridge, in Sweden, has a 100/100Mbit at half the cost. But that has nothing to do with net neutrality.

If my friend wants to access my server, my upstream channel is limited to 2 Mbit per second. This is by my choice, I can choose to host my server somewhere else, I could try to get a better link and so on, but basically, I decide for myself who, and how much I want to offer. There are sites that will flat out refuse to serve data to certain visitors, and that’s their prerogative.

However, with net neutrality removed, my site may get throttled or artificially bottlenecked to the point where people just quit visiting my site. I would have to deal with several ISP’s and possibly have to pay them a fee to remove the cap. If the site is not commercial* I may not have the funds to do that. I may not be aware that an ISP is throttling my site into oblivion, or even be offered an option to remove the cap.

Clearly, ending net neutrality is not the end of the world. Guatemala and Morroco are two examples of countries w/o net neutrality. In Morroco, the ISPs decided to block Skype, since it was competing with their (more profitable) voice service, so that might give you a hint of what’s to come. They did complain to the King when the ISPs went too far though.

Naturally, fast access to Facebook LinkedIn and Snapchat might be cheaper, and probably all you care about if you’re against NN.

With cloud-based IP video surveillance starting to become viable, this might prove to be another, unpredictable cost of the system. Some ISPs already take issue with you hosting a web server via your retail connection. And they go out of their way to make it difficult for you to do so: Changing your IP address every 4 hours and so on. This is to push you into a more expensive “business plan”, where they simply disable the script that changes your IP. I think it is safe to assume that if you’re streaming 30 MBit/s 24/7 to an Amazon data center, the ISP will eventually find a way to make you pay. And pay dearly. Once you’ve hooked your entire IP video surveillance system into the cloud, what are you going to do? Switch to another ISP? #yeahright

I guess the problem is that the ISP business model used to be to sell the same bandwidth 100 times over. Now that people are actually using the bandwidth, that model falls apart, and the ISPs need other means to make sweet sweet moolah. And that’s their nature and duty. But why cheer them on?

*In the early days, commercial activity on the Internet was banned.

 

HomeKit Flaw

https://9to5mac.com/2017/12/07/homekit-vulnerability/

Does this vulnerability shipping mean you shouldn’t trust HomeKit or smart home products going forward? The reality is bugs in software happen. They always have and pending any breakthrough in software development methods, they likely always will. The same is true for physical hardware which can be flawed and need to be recalled. The difference is software can be fixed over-the-air without a full recall.*

*Unless it’s a Chinese IP camera, then all “mistakes” are deliberate backdoors put in place by the government.

In Defence of Hikvision

Look at this nonsense!

Brian Karas reported on March 2 that he was hearing from multiple Hikvision security camera and DVR users who suddenly were locked out of their devices and had new “system” user accounts added without their permission.

Karas said the devices in question all were set up to be remotely accessible over the Internet, and were running with the default credentials (12345). Karas noted that there don’t appear to be any Hikvision devices sought out by the Mirai worm — the now open-source malware that is being used to enslave IoT devices in a botnet for launching crippling online attacks (in contrast, Dahua’s products are hugely represented in the list of systems being sought out by the Mirai worm.)

[I cut out some text from here (I’ll tell you why)]

According to Karas, Hikvision has not acknowledged an unpatched backdoor or any other equivalent weakness in its product. But on Mar. 2, the company issued a reminder to its integrator partners about the need to be updated to the latest firmware.

OK, so Brian hears that people who a) expose their IP cameras directly to the internet, and b) are using default admin credentials “suddenly were locked out of their devices”. My God, what kind of evil genius hacker is behind this, and there were new “system” user accounts!!?!

This must be the Chinese government’s work. Only a government organisation would be able to crack into IP devices with default passwords that are directly exposed to the internet.

When people got their shit “hacked”…. actually, let’s not call it hacked. Someone logged in, as admin, and changed things, so, not hacking. Someone had done something similar to mirai (which will take any script kiddie 30 minutes to write up, but Karas and Krebs pretend to not understand that). Hikvision sees this, and then reminds people to update their firmware, and as the new firmware does not allow default passwords (as far as I can tell), it seems prudent advice, and what you ought to do.

Krebs seems to want to play a part in all this “dangerous Hikvision camera” bullshit, so instead of posting a meaningful timeline, he spices things up, and injects this little tidbit (which I removed above to ensure a comprehensible timeline).

In addition, a programmer who has long written and distributed custom firmware for Hikvision devices claims he’s found a backdoor in “many popular Hikvision products that makes it possible to gain full admin access to the device,” wrote the user “Montecrypto” on the IoT forum IPcamtalk on Mar. 5. “Hikvision gets two weeks to come forward, acknowledge, and explain why the backdoor is there and when it is going to be removed. I sent them an email. If nothing changes, I will publish all details on March 20th, along with the firmware that disables the backdoor.”

OK, so on the 2nd the n00bs at IPVM and their subscribers are “hacked” by a genius hacker, who is able to guess the password and add new accounts, and then on the 5th, a guy who re-compiles the hikvision firmware discovers a vulnerability. In fact, he tells John Honovich that Hikvision has been very responsive in fixing the issue!! This seems to get lost somewhere between the sensationalist blogs (I think, because I am banned from IPVM).

How the hell do you make a connection between morons who exposes their cameras with default admin credentials, and someone discovering a bug in the validation of a reset packet (I guess that is the vulnerability, because I don’t know the details). You make that connection, if you think it will bring in more subscribers, and by extension, more filthy lucre.

Full disclosure: I am not paid in any way shape or form by Hikvision or any camera manufacturer for that matter. I receive no payment from this blog either, the ads you might see are put there by wordpress that hosts the blog, as compensation for hosting and traffic cost (and profit I guess), but I receive exactly $0.

Influencers

When Netflix* just started rolling out their Internet streaming service I used a VPN service, and streamed a wonderful movie called “Human Centipede”. It’s a surprising accurate portrayal of how things work in my industry, except the labia tua to sphincter connection is entirely voluntary, whereas the victims in the movie were not given an option.

I wonder if the toilet paper industry celebrates “influencers“?

Because, frankly, most people need IP video like they need toilet paper, and it should be just as simple to use. I’ve been to many places on God’s green earth and not once did I encounter toilet paper that wasn’t compatible, or was complicated to work with.

Yet, we “celebrate” leaders that take something that ought to be just as simple, and turn it into a nightmarish ordeal to install and maintain. If you are influential, then you must share some of the blame for the pile of shit that this industry is serving up again and again.

* It’s hard to fathom that a mail-a-DVD-business would be a major driver of the entire US economy, but there you go)