Advanced Enterprise Detection Capabilities (a multipart series)

I'd like to introduce a few concepts that can and *should* be used to augment your current enterprise detection capabilities. This series will not go over standard concepts that enterprises commonly deploy such as firewalls, IDS, proxies, host based AV, etc. These new concepts aren’t anything ground breaking, but they are, however, not too commonly thought of or implemented in the enterprise security stack.

“Part 1: Enterprise Wide Multi-AV Scanning? Well, Kind Of…"

The first concept that I am going to talk about is meant to augment your enterprises’ simple anti-virus (AV) products. In most enterprises, AV scanning technology may be implemented by network devices (web proxy or next generation firewall) and clients/endpoints (think Forefront Endpoint Protection, McAfee Endpoint Protection Suite, ClamAV, etc.). Lets not forget that AV software can also live on servers that offer network services, such as (and not limited to) mail servers (scanning all email attachments), sharepoint servers (scanning all files uploaded to sharepoint), etc.

In simplistic terms, most AV solutions work by either identifying known bad data (a sequence of bytes or code blocks) within a file or by using some form of heuristics on/against a file. The actual file (data) is needed to be scanned and sometimes not even the whole file needs to be scanned by the AV software. The concept that I am about to explain doesn’t actually scan any data within any file (at least not at this moment in time).

The concept is rather simple: Obtain a hash of a file and use that hash to identify other files (presumably malicious) of the same hash. There are online sources, such as Virustotal (VT), that allows you to submit/upload a suspected malicious file and have 40+ different AV packages scan that file for you. Quick note: VT also provides other useful analysis capabilities outside of just simple AV scanning, but most of these other capabilities will be out of scope for this concept. However, a VT capability that is in scope for this concept provides the ability to perform simple searches of the VT database (for free with limitations). This ability allows you to search the VT database with a hash of a file (MD5, SHA1, or 256) and outputs scan results of a previously scanned file with the same hash (see picture below). Keep in mind that for results to be displayed, a file with the same hash had to be uploaded to VT for scanning prior to your search.

I must speak real quickly to the effectiveness of AV in general. All AV solutions will not detect the same malicious files. To provide additional details on this, I recommend reading a report such as (http://www.mcafee.com/us/resources/reports/rp-nss-labs-corporate-exploit-protection.pdf) or you could even just reference the above picture as you'll notice some AV products detected a specific file as malicious while others did not. Another controversy regarding AV is that it is a failing detection capability (http://www.imperva.com/docs/HII_Assessing_the_Effectiveness_of_Antivirus_Solutions.pdf). I will have to agree to an extent in regards to AV's overall effectiveness, but use cases exist (such as the one I am currently speaking to) that encourage us to use the "collective" of AV products as an additional effective source of detection data. Most enterprises will use 1 or 2, maybe even 3 separate AV solutions within their enterprise. The old adage of “two heads are better than one” holds true with having more than one AV solution (to an extent). Having 40+ AV products scan a file raises the odds of a malicious file being detected.

I need to [re]iterate that this isn't an "end all be all" solution as you will only be submitting the hash of a file. If the file is modified in any way, and I mean *any* way, the hash will be different, thus the results may not be favorable. Again, this isn’t an absolute solution, its a gap filler to augment your current enterprise detection capabilities.

Below is a limited set of data points to keep in mind when utilizing a concept that only uses the hash of a file as a detection mechanism.

• A file with the same hash had to have been uploaded to VT for analysis.
• Hash lookups are much faster than uploading the actual file for full scanning.
• The full file being uploaded provides a more accurate analysis.
• No proprietary, personal, or sensitive data is ever sent to VT.
• Subscribing to VT Intelligence allows users to download files that have been uploaded for analysis.
• Using ONLY the hash as a piece of data to perform a lookup against does not help to  contribute to the security community.
• Free VT service has a limited for uses (searches/uploads) per IP per minute. An expansive pay service is available.

To finalize on this concept, you would want to automate this solution to reap all the possible benefits of this concept. As a recommendation, I would first target automating email attachment hash lookups as there will be many obstacles to overcome (ex. identifying types of files that you want hashed, how to obtain the hashes, writing custom software to perform the lookup, etc.). For brevity, I won’t go into detail on how this entire process could be architected from start to finish, but I will finish this post by mentioning some tools that could enable this concept to take shape.

• VT API https://www.virustotal.com/en/documentation/public-api/ - Provides scripted access to searching the database by hash (among other things).
• Bro IDS can be used to obtain hashes of files in near real time.
• Most full packet capture solutions (ex. Solera, OpenFPC, Netwitness, etc.) provide API’s to allow packet data extraction, thus allowing a separate tool to perform data carving and hash calculations.
• Vortex IDS - Have pcap data sent directly to a Vortex IDS instance and hash the relevant streams (saved as files).