Module Contribution

Overview

Omnibus was built so that anyone could easily write new modules for artifact types and they would be accepted into the application. As such, each module follows the same general format and makes use of many common libraries are available within lib/common.py.

To get used to the format of modules it is highly recommended to read the source code several of your favorite modules and see how they are written within an IDE or Text Editor.

Artifacts with Modules

Every time a command is executed, the argument is an artifact name (e.g., "inquest.net") or is converted into a name if the Session ID was provided. Artifact names are also a MongoDB key within the database. So everytime a command is ran, MongoDB searches for that artifact and returns it to Omnibus in dictionary format. This way every module can fully interact with the database stored object and contribute to it's fields.

Once a module is done running, it will return the entire artifact dictionary back to the primary Dispatch class. Dispatch will then check if artifact['data'] field for results from the module, and the artifact['children'] field to see if any new entries have been discovered that could be turned into artifacts themselves.

Example Module

This section will walk step-by-step through module creation, usage, and why everything that's there needs to be there :) Come along!

#!/usr/bin/env python

# since this module sends HTTP GET requests, let's import the GET request func lib/common.py
from http import get

# to print warning messages, we'll need to import the warning message func from lib/common.py
from common import warning

# lastly, since this particular module requires an API key we'll need the get_apikey function
# from lib/common.py as well
from common import get_apikey

# each module must contain a class named Plugin
class Plugin(object):
    
    def __init__(self, artifact):
    # __init__ only ever accepts one parameter, the artifact dictionary sent from Dispatch
    
        # to interact with the artifact, it needs to be assigned to self.artifact so other functions can use it
        self.artifact = artifact

        # module results are stored inside the artifacts "data" dictionary,
        # since we yet don't know if the modules name is a key in this dictionary we first create it
        self.artifact['data']['clearbit'] = None

        # assign the retrieved API key so other functions can use it
        self.api_key = get_apikey('clearbit')


    # every Plugin() class must have a "run" function. this is what's called to start the actual modules task
    def run(self):
        # here we create a URL for Clearbit.com using the value of self.artifact['name'] as the format string
        url = 'https://person.clearbit.com/v1/people/email/%s' % self.artifact['name']
        
        # here's were the self.api_key comes into play
        # this specific API endpoint requires an Authorized HTTP header with the self.api_key value inside
        # pre-pended by the "Bearer " string
        headers = {
            'Authorization': 'Bearer %s' % self.api_key,
            'User-Agent': 'OSINT Omnibus (https://github.com/InQuest/Omnibus)'
        }
        # @note: it is good practice to add this user agent you see whenever an HTTP request is sent
        # so services know where their traffic is coming from

        # placing the http.get() call inside a Try/Except gives a chance to make sure we don't raise
        # any unhandled exceptions that might break any given module and therefore the entire Console sesssion
        try:
            # sending the GET requests to the URL with our headers,
            # we get back status (the "ok" response from the requests library)
            # and the full response
            status, response = get(url, headers=headers)

            # we only can act on results in "status" is True, since anything else means the HTTP request
            # failed
            if status:
                
                # perform some sanity checks to make sure our data is in the response
                if 'error' in response.content and 'queued' in response.content:
                
                    # if it's not then lets print a Warning message gracefully without raising any real error
                    warning('results are queued by Clearbit. please re-run module after 5-10 minutes.')
                
                else:
                    # since Omnibus handles raw JSON data, we can safely assign the response.json() results
                    # to the "fullcontact" dictionary key we created earlier
                    self.artifact['data']['fullcontact'] = response.json()

        except:
            pass


# the "main" function must exist in all modules
# Dispatch.run() calls the "main" function of each module
def main(artifact):
    # Dispatch passes the entire artifact dictionary to each module so it can be updated with module results

    # first lets create out Plugin object, passing the artifact as it's only parameter
    plugin = Plugin(artifact)
    # next call plugin.run() to perform the actual work behind the plugin
    plugin.run()

    # if all goes according to plan, the returned artifact will now have JSON content inside of:
    # artifact['data']['fullcontact']
    # this data is then sent back to the user for filtering, viewing, etc.
    return plugin.artifact