Published : 7 years ago, on 03/03/2018

By Anthony Perridge, Regional Director, ThreatQuotient

Research shows that the number of connected devices will grow from 28 billion today to an expected 50 billion by 2020. Meanwhile the number of connected people will increase from 2 billion today to around 6 billion by 2020. This means that both the network attack surface and the human attack surface will expand.

On top of that there is a cybersecurity hiring crisis. It is expected that there will be 1.5 million people missing from the security workplace by 2019. This naturally means with fewer defenders and more opportunities for adversaries, incidents and the associated data will also continue to rise.

Based on the average dwell time before a breach is discovered and the average number of reported incidents, at a bare minimum we’re talking about 328 concurrent ongoing breaches at this very moment. While most adversaries are financially motivated (stealing information such as credit card numbers, medical records and proprietary information that can be sold on the dark web), other motivation includes hacktivism, cyber warfare and cyber espionage.

So how can we effectively look into large datasets from security incidents and help detect and prevent the next attack?

The need for data mining

Each incident report includes hundreds of indicators of compromise (IOCs). IOCs can be related to the victim’s host evidence (such as malware type, file name, hash file and registry keys). Additionally, IOCs can be related to the communication lines to the malicious link (such as IP address, domain name, URL and port numbers). Both host-based and network-based IOCs indicate a potential intrusion in your network.

The challenge is that the data is not always linked and it’s a massive amount to sift through. If you have 4 threat intelligence sources that provide even just 300 indicators a day, that means you’re getting at least 500,000 indicators a year! Organisations do not have time to investigate them all and pushing that information to your sensor grid (IPS, firewalls, etc.) isn’t practical. You end up with tons of false positives and poor performance.

There are many threat intelligence providers out there – from open source to commercial to industry-specific feeds – that continuously update a list of IOCs. Each provider adds a little piece to the puzzle in order to help illustrate as best as possible the current threat landscape. This has led to the industry adopting threat intelligence platforms to help bring all the pieces together into one repository for a single picture of the threat.  The threat intelligence platform automates the ingestion, correlation, normalization and de-duplication and serves as a single source of truth for all teams and systems within the organisation.  It becomes the tool to mine data to understand threats; add context to analyse and investigate; and effectively use the intelligence within your organisations processes and tools.

Data mining techniques

Classification helps reduce the noise. Examples can include IP address, domain name and URL. You can also classify by using attributes for example:

• Malware family
• Geography
• Language

Classifying by adversary allows you to look at attacks focused on your industry and infrastructure. Whilst classification by incidents or events (age, owner, day of week, user ID) allows you to connect the dots across the kill chain. Lastly you can classify by relevance, sorting by CVE, OS, user or brand of device – so you get an even more focused look at what you should care about.

Once threat data is classified, you also need to prioritise because not all data are equally relevant to every organisation. Using an applicable threat intelligence platform can help you score and automatically prioritise threats.  One key consideration, however, is that every organisation is different in terms of their business, security operations and risk profile. This requires scoring and prioritisation based on parameters you set, including indicator types (IP address, malware type, host-based vs network-based, etc.) and indicator source (open source, commercial, industry based, as well as internal sources like your SIEM and ticketing systems).

Threat intel investigations

Context is very important in understanding threats, but unfortunately, we don’t always get all the context we need so we need to use enrichment tools.

Some of the methods we use are similar to the process in the TV show, “Who Wants to be a Millionaire?” where you use your lifelines to get help. For example, asking the audience a question is, in effect, crowdsourcing and using VirusTotal is a good example of this in our world. Or you might phone a friend when you have a specific question you know they can answer. This correlates to contacting a specific vendor that specialises in that type of attack for assistance.

Another method used in threat investigations is link analysis which involves identifying the relationships between bad actors, transactions, objects, servers, IP addresses, and specific malware families.

Whatever method you use, the human element is very important. You need a person and their human intelligence working in concert with the technology and tools. A threat intelligence platform is designed to facilitate this with centralised intelligence sharing, analysis and investigation.

Threat intel effectiveness

So how can we measure the effectiveness of threat intel?  Effective threat intelligence examines total attack surface, taking into account the industry, location, internal software and networks, vulnerabilities, physical threats to personnel and property, third-party vendors, brand reputation risks, and customer goodwill. It comes down to reducing time to detection and time to respond.  If we can reduce these numbers, then we’ve done our job.

Conclusion

Next year the threat intelligence market is expected to exceed over £1 billion. I believe that the role of threat intelligence is to support the entire security and defence strategy. Everyday organisations are blindsided by cyber-attacks and analyst risk missing external threats that can impact on the business. Threat intelligence is empowering these organisations to develop a proactive approach to cyber security. With over 5000 new vulnerabilities per year and 400 million malware variants also popping up annually, threat intelligence numbers are showing that organisations need to be on the front foot and proactive in their engagements for the future and beyond.