Holmes Processing

Publications

Scientific Publications & Industrial Conferences

Description

Holmes Processing is developed in partnership with academia and industry. To assist our academic partners, please cite or provide an acknowledgement to their work. It is their currency and helps them greatly.

Scientific Publications

Title	Publication	Links
SKALD: A Scalable Architecture for Feature Extraction, Multi-User Analysis, and Real-Time Information Sharing	19th International Conference on Information Security (ISC)	bibtex
Adaptive Semantics-Aware Malware Classification	13th Conference on Detection of Intrusions and Malware & Vulnerability Assessment (DIMVA)	bibtex
Distributed Services for Large Scale Dynamic Malware Analysis	Thesis	bibtex
Large Scale Anomaly Detection Using Spark	Thesis
Automated Detection of Malware Relationships	Thesis

Presentations

Title	Conference	Links
Internet-Scale File Analysis	Black Hat - USA 2015	bibtex
A Study of the Rich Header and Respective Malware Triage	Hacktivity - 2016
From Mole Hills To Mountains - Revealing Rich Header And Malware Triage	RSA Conference - 2017

Gateway

Tasking Validation and Routing

Overview

Main program for receiving tasking and objects. It validates input, checks authentication, and pushes the requests to the pipeline. Holmes-Gateway consists of two components: The Master-Gateway and the Slave-Gateway (also known as Organizational Gateway). Holmes-Gateway is meant to prevent a user from directly connecting to Holmes-Storage or RabbitMQ. Instead tasking-requests and object upload pass through Holmes-Gateway, which performs validity checking, enforces ACL, and forwards the requests.

A user always connects to the Master-Gateway of their own organization.

If the user wants to upload samples, he sends the request to /samples/ along with his credentials, and the request will be forwarded to storage.

If the user wants to task the system, he sends the request to /task/ along with his credentials. The Master-Gateway will parse the submitted tasks and find partnering organizations (or the user’s own organization) which have access to the sources that are specified by the tasks. It will then forward the tasking-requests to the corresponding Slave-Gateways and these will check the task and forward it to to their Rabbit queues. Slave-Gateways can be configured to only accept requests for certain services for certain organizations and to push different services into different queues.

This way Slave-Gateway can push long-lasting tasks (usually those that perform dynamic analysis) into different queues than quick tasks and thus distribute those tasks among different instances of Holmes-Totem and Holmes-Totem-Dynamic.

Highlights

Collaborative tasking: Holmes-Gateway allows organizations to enable other organizations to execute analysis-tasks on their samples without actually give them access to these samples.
ACL enforcement: Users who want to submit tasks or new objects need to authenticate before they can do so. Also an organization can decide which services an other organization is allowed to execute on their samples.
Central point for tasking and sample upload: Without Holmes-Gateway, a user who wants to task the system needs access to RabbitMQ, while a user who wants to upload samples needs access to Holmes-Storage.

Code: Github Repository
Documentation: Read the Docs
License: Apache 2

Totem

Static Analysis and Queries

Overview

The Holmes-Totem Planner is responsible for turning data into information by performing feature extraction against submitted objects. When tasked, Holmes-Totem schedules the execution of its Services which are capable of performing static and dynamic analysis as well as gather data from third parties.

The Holmes-Totem Investigation Planner is optimized for executing extraction Services that complete in a few seconds, i.e. static analysis and 3rd party queries. When dealing with Services that take longer to complete, we recommend pairing the Holmes-Totem Planner with Holmes-Totem-Dynamic

Highlights

Holmes-Totem is pre-packaged with with a number of static analysis and query Services to help you get started. A few highlights:

Name	Object Type	Description
ASNMeta	IPv4/6	Collects ASN information from Team Cymru
DNSMeta	Domain	Collects current DNS information from your local resolver and authoritative NS
PassiveTotal	IPv4/6, Domain, Email	Returns available Passive Total information
GoGadget	Binary Executable	Extracts the major gadgets of an executable
Objdump	Binary Executables	Provides a parsed version of the objdump output
PEiD	File	Provides PEiD information (packer, compiler, etc) for files
PEInfo	PE32	Custom version of CRITs’ PEInfo. Includes Rich Header parsing
RichHeader	PE32	Rich Header extraction for PE32 files
VirusTotal	Binary	Returns available VirusTotal information with a public or private key
Yara	File	Performs Yara signature matching with a default rule pack or custom rule
ZipMeta	Zip	Provides meta information contained in zip files

Code: Github Repository
Documentation: Read the Docs
License: Apache 2

Totem-Dynamic

Dynamic Analysis

Overview

Just like Holmes-Totem the “Dynamic” Planner is responsible for turning data into information by performing feature extraction against submitted objects. When tasked, Holmes-Totem-Dynamic schedules the execution of its services which are focused on dynamic and other long or indefinite running analysis tasks.

Highlights

Inner workings

Totem-Dynamic is based on distributed_cuckoo and uses a similar concept. Just like Totem it accepts incoming tasks via AMQP and sends the results back via AMQP as soon as they are availible. Internally the application is mostly devided in three parts. feed is accepting new tasks from the outside, checks the status of the requested service and - if everything is fine - submits the task to the service or else postpones it until the service status is good again. check then periodically checks if the service is done with the task or if it still need time or if an error occured. submit then collects the results from the service and sends it out

Services

Name	Object Type	Description
cuckoo	Binary Executable	Performs a cuckoo analysis on the given binaries
VirusTotal PLANNED	Binary Executable, IPv4/6, Domain	Returns available VirusTotal information with a public or private key

Code: Github Repository
Documentation: Read the Docs
License: Apache 2

Overview

Extract

Store

Analyze

Pillars

Computer security threats are every growing, constantly evolving, and actors are deliberately attempting to thwart security practices. To counter this, Holmes Processing Architecture is based upon 3 core pillars.

Resilient

Scalable

Flexible

Projects