ShowcaseInfo.json Generation

Adding content in SSE is easy. For end users, you can run everything via the GUI. For partners, and SSE authors, you do need to interact with the JSON a bit.

Schema

The schema is thoroughly documented in the dedicated SSE Schema section.

Architecture

The beating heart of SSE is ShowcaseInfo.json. This is what contains almost all of the data that is of use to SSE. At the core of nearly every SSE dashboard is either a call to the rest endpoint /services/SSEShowcaseInfo or a Splunk search for | sseanalytics which immediately calls that rest endpoint on the back end.

The rest endpoint is defined as such:

web.conf:

[expose:SSEShowcaseInfo]
methods = GET,POST
pattern = SSEShowcaseInfo

restmap.conf:

[script:sseshowcaseinfo]
match                 = /SSEShowcaseInfo
script                = generateShowcaseInfo.py
scripttype            = persist
handler               = generateShowcaseInfo.ShowcaseInfo
requireAuthentication = true
output_modes          = json
passPayload           = true
passHttpHeaders       = true
passHttpCookies       = true

When generateShowcaseInfo.py is called, it then uses the URL to pull out configuration parameters, and then walks through the steps below to pull in the core ShowcaseInfo.json, add in any custom content, and enrich it with a bunch of files and lookups.

Steps

Initialize Connections

Initialize the service object that will be used by API calls. It also pulls the URL for splunkd in case the kvstore API is failing and it needs to fallback to a GET to the API directly. (Rare, but sometimes kvstore can be finicky.)

Checking Enabled Content

Pull all of the contents of essentials_updates.conf via the API (aggregated across all apps). Then use that to pull out which apps are disabled vs enabled.

Pulling Updated MITRE ATT&CK

We have a kvstore dedicated to storing large JSON files. That is handled via pullJSON, another rest endpoint. pullJSON supports localization (not relevant for MITRE), and has the ability to grab the file if the data isn’t in the kvstore.

The overall flow for updating these kvstore items is (as described in Partner Integration - Posting Content Online, which uses a similar method) that the user’s web browser will ask the kvstore when the last MITRE check was. If it’s been over a day, it will then go and look for an updated version of MITRE – if it finds it, it will stash it in the kvstore and pullJSON will render that file instead of whatever is on the file system.

Pulling SSE kvstores

Simple Pulls

The following kvstore collections are pulled in with no modifications:

1. bookmark
2. local_search_mappings
3. data_source_check

Custom Logic

The following have minimal processing:

1. custom_content - inputs are cleaned as described in Partner Integration - Security and Data Cleaning
2. data_inventory_products - the inputs are pulled in and tossed into an array of scores per DSC, so that they can be easily combined at the product level (where there may be multiple DSCs). Also the DSC to productId match is created.

Grab Core Files

ShowcaseInfo.json

ShowcaseInfo.json is all the shipped content in SSE.

Search Builders

Each of the search builder files are pulled in and tossed into a file. If there were naming conflicts (e.g., “Detect Bad - Live” in two different JSON files) it will silently override, but that shouldn’t happen.

Other Supporting Files

There are a variety of other supporting files that are pulled in for enrichment as well:

• data_inventory.json
• MITRE ATT&CK and MITRE Pre-ATT&CK – these will actually never be grabbed directly (this code should be rationalized out) as the pullJSON call above will never fail to deliver some MITRE content.

Enrich Custom Content

Next we add the custom content in to the main ShowcaseInfo as if we grabbed it from the JSON file itself.

MITRE Parsing

Grab the MITRE configurations and spin them into a variety of objects for usage later. Parsing out MITRE ATT&CK from the JSON is a bit of a pain. The short version of it, for the purposes that SSE cares about:

• Tactics are objects of type x-mitre-tactic
• Techniques are objects of type attack-pattern
• Groups are objects of type intrusion-set

All of these are related to each other through objects of type relationship and relationship_type uses. All objects have an ID, which maps to the source_ref and target_ref fields.

The implementation of parsing this out in SSE is probably not ideal (it feels like a classic CompSci exam question) but it works and is fast enough.

Clear out invalid characters

There should never be invalid characters in an ID, but if there are it would break things downstream, so let’s clear them out.

Enrichment and Processing

A variety of enrichments occur here. It’s frankly easiest just to read the code, as they’re mostly tactical.

Enrich with Searches

This step looks through all of the search builder JSONs (showcase_simple_search.json etc.). It then goes through all of the Showcases in ShowcaseInfo.json, for each with any examples it goes through each example and looks for it in the list of all the search builder JSONs. if there is a match, it adds that as ShowcaseInfo['summaries']['my_summary_id']['examples'][NUM]['showcase'] = my_search_builder_obj

Local Search Mappings

This checks each piece of content against the list of savedsearch.conf mappings in local_search_mappings and then sets the search_title to the equivalent search. (Known Limitation: only one search can be provided per piece of SSE content. Customers with multiple searches that are similar to the same object would need to create custom content in SSE to complete the mapping. The UI doesn’t explicitly prevent this behavior.)

Data Availability Enrichment

Here we take each piece of content and look it up in the rendered Data Availability matrix to add the fields data_available and data_available_numeric.

MITRE Enrichment

Here we take the MITRE ATT&CK Framework variables (parsed above) and enrich each piece of content with the:

• Tactic and Technique Name
• Technique Description
• Which MITRE Matrix it came from (ATT&CK, Pre-ATT&CK)
• Threat Groups
• Search Keywords (e.g., mimikatz)

Defaulting Missing Fields

Many parts of SSE expect certain fields to be present, and while those fields generally are, we don’t want downstream code to fail due to unreliable content configuration. So for simplicity, we have a variety of ways to clean out content, from setting it to an empty string, setting the string “None”, etc. Those are all configured here.

Excluding Disabled Channels

This will iterate through all of the content in ShowcaseInfo, then check to see if the channel is in the exclusions list (channel_exclusion[...]) and pull the content if it should be excluded. This can be overridden by supplying ignoreChannelExclusion=true in the request.

Exporting to sse_content_exported

There is a lookup (sse_content_exported) used for mapping savedsearch.conf names to the metadata inside of SSE. Whenever you run ShowcaseInfo it will check to make sure the content in that lookup is correct, and fix it if not. The fields included in that export are listed in the fields = [...] line at the top of this section, and are mirrored in collections.conf and transforms.conf.

Optional Minifying

A few pages (particularly the main Security Contents page) can request a Mini version of ShowcaseInfo by adding fields=mini to the URL. This strips out all the fields not specified in mini_fields and dramatically reduces the amount of data transferred back. The goal here is to make it so that the user doesn’t have to download 1+ MB over a bad internet connection. (There was at one point a series of bugs that ballooned that to multiple megabytes in size – but while analyzing that bug, the option to shrink down the file volume was added.) As of this writing, the raw JSON downloaded is about 1.2 MB which is automatically compressed by Splunk down to 202 KB of network.

Return

Data is returned in a JSON format.

Error Handling

At no point should ShowcaseInfo return an error (this was the case in early versions of the rest endpoint, but that was problematic). Instead, there are two elements added into the SSE output, debug and throwError.

throwError will fire if there is a critical error – effectively if an exception is caught on an important step in this process (which is almost all the steps). The debug object will contain a variety of debugging information (particularly: timing checks), and any exceptions will also be added into this debug.

From the Security Contents page, the ShowcaseInfo response is added to the window object, so you can open up the JavaScript console and run:

console.log("throwError Status", ShowcaseInfo.throwError);
console.log("debug Status", ShowcaseInfo.debug);

throwError will hopefully be false, and here is an (abridged) output fo the debug status:

[
    "Stage -5 Time Check:9.05990600586e-06",
    "Stage -4 Time Check:8.20159912109e-05",
    "Stage -3 Time Check:9.20295715332e-05",
    "Stage -4 Time Check:0.000141143798828",
    {
        "localecheck": ""
    },
    "Not going cached! Prepare for a long ride.",
    "Stage -1 Time Check:0.000144004821777",
    {
        "channel_exclusion": {
            "mitrepreattack": false,
            "custom": false,
            "Splunk_App_for_Enterprise_Security": false,
            "mitreattack": false,
            "Enterprise_Security_Content_Update": false,
            "Splunk_Security_Essentials": false,
            "Splunk_User_Behavior_Analytics": false
        },
        "override": false,
        "msg": "Final Channel Exclusion"
    },
    "Stage 0 Time Check:0.106211185455",
    "Stage 1 Time Check:1.20219802856",
    "Stage 2 Time Check:1.27426409721",
    {
        "store": "bookmark",
        "message": "I got a kvstore request"
    },
    "Stage 3 Time Check:1.29301118851",
    {
        "store": "local_search_mappings",
        "message": "I got a kvstore request"
    },
    "... abridged ...",
    "Stage 25 Time Check:1.62445807457"
]

Accessing via JavaScript

From a Splunk dashboard, running the following in the browser will get you the resulting object:

require(['json!' + $C['SPLUNKD_PATH'] + '/services/SSEShowcaseInfo?bust=' + Math.random()], function(showcase){
    window.debug_showcase = showcase; 
    console.log("Got Showcase", showcase); 
})

Accessing via Splunk Search

You can access the output of all of the summaries via the | sseanalytics command. It helpfully breaks out the key fields separately, and automatically converts the inline-multi-value pipe-separated fields into native Splunk multi-value fields. You can also ask for the full JSON output, identical to what is shared when doing a direct REST call.

| sseanalytics include_json=true | search channel=ButtercupLabs | table id summaries *