The Packet Never Lies

May 24, 2022 | Vantage Point

Blog

The Packet Never Lies

The Black Box

“Self-learning, autonomous artificial intelligence (AI) security solution.”

That’s the marketing double-speak to sell you a “black box” that supposedly teaches itself about your computer network. Then, it autonomously spots bad stuff happening and takes actions to fix it.

It sees all things, knows all things, and is never wrong. You don’t need to do anything. Plus, it has an awesome graphical user interface (GUI).

Well, not exactly. You see, it may learn bad behaviors and think they’re good. It has to be trained properly. It sounds like something from a movie I once saw.

HAL 9000 from 2001: A Space Odyssey

Customer Story

In the not-too-distant past, a client called us to help assess their security posture as part of a merger and acquisition (M&A) transaction. The company is part of an international manufacturing organization. They had a large footprint of internet of things (IoT) manufacturing devices and a network interconnected with global supply chain partners. The internal security team was pretty confident. They greeted us with folded arms. After all, they’ve had one of those “black boxes” monitoring their environment for several years. There is no way we would find anything.

We deployed our PacketWatch sensors and began to collect data. Right away, we detected an older “bot” operating on one of the computer controllers for an IoT device. “No way,” they said. It’s simply not possible. And they dismissed our claim. We showed them in our PacketWatch analysis GUI, but they still didn’t believe us. So, we pulled a packet capture (PCAP) of the device’s traffic over the previous 48-hour period and showed the internal team the results.

The packets never lie. Sure enough. There it was.

Wireshark was brought in to definitively prove our claim. We had all the packets, including the payloads showing repeated malicious activity: inbound commands and outbound responses. If we only had net flow data, they probably would have wiggled off the hook claiming some ambiguity in interpreting the data. But we had the whole enchilada—full packet capture history. If we didn’t have historical data, a quick fix would have caused the bot to “disappear” before additional forensics could be run. But there’s no wiggle-room when you have the actual packets. In fairness, we asked the black box for its opinion, but it wasn’t capable of responding.

That meeting prompted an angry call from the client’s Chief Information Security Officer (CISO) to the black box vendor demanding answers. He had staked his reputation on the black box. The vendor went back and reviewed their application history. They couldn’t say exactly how, but the “bot” traffic was detected and somehow whitelisted. What? It had been whitelisted several years prior. So, a detection went without action, and the black box had “learned” it was OK. Oops. How do you un-learn that?

As a result of our findings, the client had to send an embarrassing incident disclosure to their supply-chain partners. The client’s CEO was angry, too. He had blown his budget on all this black box stuff the CISO guaranteed would work. The truth was that an entry-level human analyst with the proper tools would have found the bot easily for roughly the same spend.

Would this be a resume-generating event for the CISO?

Fortunately, not. As we worked together on the balance of the assignment, we continued to show our value.

We were able to troubleshoot an application configuration error by providing them with the session negotiation packets. We showed them exactly where the handshake was failing. None of their other tools could.
We also showed them some misconfigured DNS entries creating daily internal DNS storms.
Our threat hunters showed them several design vulnerabilities (i.e., clear text credentials) that needed attention.

Again, the packets never lie. The CISO was now a hero for bringing us in.

Humans are Necessary

The point here is not to disparage the black box but to convince people that experienced “humans” are necessary to the security process. A black box can automate the detection of threats, but the only sure way to adjudicate a threat is for a human to go back to review the actual packets. If you don’t have the packets, that’s a problem. If your team lacks network visibility at the packet level and/or needs help figuring out exactly what your black box solution has been doing all these years, please call us. We’d love to help you out.

Tags:

Happy Anniversary, Still WannaCry.

May 16, 2022 | Vantage Point

Blog

Happy Anniversary, Still WannaCry.

On Friday afternoon, May 12th, 2017, we started to hear about WannaCry ransomware which would ultimately impact over 250,000 computers worldwide. WannaCry, Eternal Blue, Shadow Brokers, and Server Message Block (SMB) exploits seem so long ago. What have we learned in those 5 years? Not enough, apparently.

Although the patch that would protect against WannaCry was issued by Microsoft on March 14th of that same year, it seems many organizations didn’t get around to installing it in time. Exploiting an SMB vulnerability efficiently abused by the NSA for years and then leaked to the public by the Shadow Brokers certainly caught people off guard. Ransomware has only accelerated from there. Some say we may have reached “peak ransomware” last year.

All these years later, organizations are still struggling to patch vulnerabilities before exploits take advantage of them. Or, unfortunately, they patch after the exploit has been utilized and never checked to see. You’re lucky to have one day to patch a critical vulnerability nowadays.

Over the past 5 years, businesses have purchased more security tools and bought cyber insurance policies in the hopes of mitigating costs associated with their accumulated technical debt. Gone is the thought that some “black box” artificial intelligence (AI) machine can solve all of your security problems without you having to do anything.

Cyber insurers have lost their rears and are pushing back — jacking premiums, cutting coverages, and low-balling recovery efforts. Insurers also started asking more questions in a vain attempt to better underwrite risks. Too late.

Even the SEC has stepped up its efforts to force companies to better disclose their cyber practices and risks to investors. Upstream supply chain partners are asking what you do to mitigate risks. It’s not just “are you doing something” any longer. It’s now “Are you doing the right things?”

So, what are the right things?

You need to practice good cyber hygiene; identify and patch vulnerabilities; have a strong, resilient infrastructure; verified security controls; and well-rehearsed IR plans.
You need an experienced security team with visibility over the network and hosts (not just hosts). Logs are great but come after the fact.
You need battle-hardened humans that actually hunt for badness in your network before it gets you. Not some automated detection tool whose “check-engine light” goes on when the oil is low. You need real people looking for signs that other real people are exploiting weaknesses in your systems.
You need to test your controls and procedures to see how effective they actually are.

With all due respect to your IT team, they just won’t measure up against an adversary funded by a foreign government.

What does targeted adversary activity look like in your environment?
Has your team ever experienced that?

Businesspeople are sick of the slick marketing hype from security vendors making exaggerated promises. Organizations need real solutions from seriously experienced people.

I hope that by the next anniversary of WannaCry, more people will have implemented the elements described above with internal resources or with a team like ours—hunting every day to identify and close security gaps.

If you need help, we are here and ready.

Tags: