Partner Integration

Table of Contents

Goals of Integrating Content

Splunk Security Essentials provides a wide variety of tooling to help users discover what content they should adopt in the world of security for Splunk. They can analyze their content against MITRE ATT&CK, they can filter for what they’re able to do at no cost with existing data, they can mark the content that is enabled, they’re able to see what they can do with premium solutions, and they can track and report on their success.

The app also supports Splunk’s partner ecosystem. If you are a commercial security partner, or even an open source provider, you can easily add your content into Splunk Security Essentials. Users will then be able to track what content they have, and showcase how your content helps them meet their needs. This benefits the users by allowing them to have a full view of all the things they’re able to do to improve their security posture, and also benefits you by showcasing the benefits of your technology. For companies that normally restrict access to their analytics, making some freely available can also provide more visibility and drive more users to your company.

End User Experience

Finding Your Add-on

Content is not automatically enabled in SSE – users must first install your add-on to SSE. Your add-on can be served through an existing app that you already provide to your users, or you can create a custom-built add-on specifically for Splunk Security Essentials that you post to SplunkBase. Or both – ultimately you want to provide an essentials_updates.conf, and you can use either method.

There are three methods through which users will discover new content in Splunk Security Essentials:

Searching on SplunkBase

Since your add-on is installed on SplunkBase, it can be found through normal search. In addition to when searching for your name, it will also show up when users search for Splunk Security Essentials (presuming that you put “Splunk Security Essentials” in your app name or description).

SSE Website

From the Splunk Security Essentials website, there is a Partners page that contains the list of partners, with links to their SplunkBase content.

1. From the main website, users can click on Partners.

   

2. They’ll then be greeted by an (alphabetical order) list of partners.

   

In-app Link

SSE also has a link on the Content Sources configuration to the SSE Website.

1. From the configuration menu, they will see the partner link.

   

2. They’ll then see the same partner list on the SSE website shown above.

Through Your Existing App

If you already publish an app, you can add an essentials_updates.conf to that app and it will automatically plug into Splunk Security Essentials.

Enabling Your Content

1. Users will start by installing your add-on.

   

2. Your content will then be automatically downloaded the next time the user opens Splunk Security Essentials.

   

3. Your content will show up throughout the app, in all dashboards, just as if it was produced by Splunk in the normal build.

   

4. Users who click on your content will be able to view all the details, and see your logo at the top of the page. The logo will be a link to the URL you provide.

   

5. The detailed description of your organization will also be present, and available to users. Your logo will show up again (also linked), along with the full description, and a link to learn more (also to the URL you provided.)

   

Process Overview

The process for putting your content into Splunk Security Essentials is straightforward. You must first convert your content into the Splunk Security Essentials format, then post it somewhere accessible to be downloaded, and finally post an app to SplunkBase that will point Splunk Security Essentials to this file.

Converting Content

The custom content dashboard in Splunk Security Essentials makes this easy (though you should expect a small amount of tweaking of the JSON – this is generally trivial). You can also use the Correlation Search Introspection and Search Mapping to guide you through the creation of all of your content if your content lives as active searches in savedsearches.conf. If your content lives in another repository (such as a database in your code repository) you can always build custom code to convert that into the SSE format – see Content Format below.

Posting Content Online

Splunk Security Essentials downloads new content through the user’s browser (no requirements for proxy configurations on Splunk itself). On every page load, the user’s browser asks Splunk (external_content_lookup) when the content was last checked. If it’s been more than 24 hours, it will check to see if the content is up to date.

If there is a build_url and build_field configured, it will first do a GET of build_url, expecting a JSON object, and look for the buildnum. If the buildnum != the last buildnum received (again, in external_content_lookup) it will proceed download the full content.

If there is no build_url, it will also proceed to download the full content.

To download the full content, it will perform a GET of the content_download_url, expecting JSON object with all of the content (see Content Format for more).

It’s generally easiest to store this content in github of an S3 bucket. If using github, make sure that you use the raw URL format. If using S3, make sure that you configure your bucket to enable Cross-Origin Resource Sharing (CORS). See the following URLs for advice on configuring CORS in S3: 1. How Do I Allow Cross-Domain Resource Sharing with CORS? 1. StackOverflow Example

Post App to SplunkBase

The last step in the process is to tell SSE where to grab your file. This is done by creating a new stanza in a file called essentials_updates.conf. Here’s an example configuration:

default/essentials_updates.conf

[ButtercupLabs]
channel=ButtercupLabs
name=Buttercup Labs
description=Buttercup Labs produces high quality security analytics run through Splunk Enterprise. While Buttercup sells a commercial threat analytics app, they have also released a variety of community content. All is available through Splunk Security Essentials.
type=app
app_context=Splunk_Security_Essentials
content_download_url=https://go.splunksecurityessentials.com/myContentLocation

The [StanzaName] should be unique to your org (but won’t be referenced by anything else). The channel will be configured on the back end, but won’t impact your users. It is recommended that these two fields match, and spaces are not allowed in either.

The name field will appear in many places throughout the app. It is generally expected that this be the name of your company (or your group or your online identity, etc.). It will show up in the filters on the Security Content page, and then when users view your actual content. The Description will show up only on the app configuration, where users enable or disable different content sources.

type should always be app.

app_context should always be Splunk_Security_Essentials (it is built to be able to support the other Essentials apps, but the other Essentials apps do not support this functionality).

Important: When testing in your own environment, make sure to restart Splunk after making changes to essentials_updates.conf, so that Splunk will re-read this file.

Make sure that you also create an entry in metadata/default.meta in your environment. By default, Splunk doesn’t share configurations to all systems in the app, but default.meta allows us to set that configuration. (Splunk admins are very familiar with sharing configurations to all apps, but why make them?)

metadata/default.meta (Add to the bottom of the default configuration)

[essentials_updates]
export = global

Avoiding a Splunk Restart

While when developing you will need to restart Splunk, to convince the system to recognize the new configurations, there is no need for users who install your app to restart Splunk. However, we need to tell Splunk that this is the case. We do this by adding the following configuration to your app.conf.

[triggers]
reload.essentials_updates = simple

Example App

Here is a working example of the (standard) SplunkBase deployment method, here.

Offline (Local) Method

If you do not want to provide an online auto-update of content, but rather ship content in your app, you can simplify your deployment somewhat. You will not need a web server, and can instead ship the JSON file in your app itself via the appserver/static folder.

To use this method:

1. Create an appserver/static folder (if it doesn’t already exist) and put your json file in that directory
2. In your default/essentials_updates.conf file, tell SSE to look locally by starting your line with SPLUNKD, as shown below.
3. Make sure to apply the default.meta and app.conf settings prescribed above.

[ButtercupOfflineContent]
channel=ButtercupOfflineContent
order=10
name=Buttercup Offline Content
type=app
app_context=Splunk_Security_Essentials
content_download_url=SPLUNKD/static/app/Buttercup_App/mycontent.json

No naming scheme applies to your app. (We use SA-SSE-Custom in the sample below, but it is only convention without any requirement.)

Example App

A working example of this model can be seen in the example app, here.

Content Format

SSE’s ShowcaseInfo expects a very specific format. For detail on this, visit the schema page: SSE Schema. For the purposes of partner integration, not all fields are fully supported. It’s generally best to focus on the subset of features detailed further down on this page, under the heading Security and Data Cleaing.

Using Custom Content in Splunk Security Essentials

1. Start by opening the Custom Content dashboard in SSE.
2. Click Add Custom Content, from the gray box in the upper right hand side.

3. You will be greeted by a series of options – these are almost all of the same options that are in place for standard content in Splunk Security Essentials, allowing you to fully represent your content. The items at the start are required – some may be less relevant for your organization, but there are components of the app that expect that they are set.

   

4. You can expand the accordion menus below to show additional configuration options. You will generally want to configure a few options in Metadata fields, such as your company / organization name, description, link and logo, and useful metadata such as the Kill Chain phase and MITRE ATT&CK. Descriptive fields includes optional details such as How to Implement and Known False Positives. If you wish to provide an actual search string, the Search menu will enable that.

   

5. To provide a good user experience to the user, ensure that you provide your company information. The description does not allow HTML or Markdown, but if you enter a \n it will automatically be converted to a line break.

   

6. Once you save this, the configuration is added into the custom_content_lookup kvstore collection. You can pull the JSON file from this location, which has everything you’ve configured.

   

7. Important: You must adjust this file slightly. You will want to add in the channel (configured in your essentials_updates.conf file) and the id to this configuration when you migrate it to the final hosted file. You will likely want to change the ID to indicate that it is not custom content, but rather something from your organization – make sure that you update the link in the dashboard attribute as well.

Custom Fields Available

There are a few custom fields available to better display content for your users:

Company Info

As detailed above, it’s recommended to have details on your company information. You can add fields like the following:

{
    "company_description": "Buttercup Labs is the premier distributor of Pony-related security analytics. We have been protecting organizations from bad ponies for over ten years now.\n\\n\\n\nEnjoy our freely available content for detecting bad ponies in your environment, and reach out to us for a demo or trial license of our premium Pony Detection app!\n\\n\\n\nHave you successfully found bad ponies in your own environment? Buttercup Labs is hiring! We are a wholly owns subsidiary of Buttercup Games.",
    "company_link": "http://buttercupgames.com/",
    "company_logo": "https://image.slidesharecdn.com/splunklivesfhowtoalignyourdailysplunkactivitiesbreakoutsession-160317192319/95/how-to-align-your-daily-splunk-activities-breakout-session-23-638.jpg?cb=1458242654",
    "company_logo_height": 250,
    "company_logo_width": 444,
    "company_name": "Buttercup Labs"
}

Additional Context

SSE includes hardcoded fields for a Search, Known False Positives, How to Implement, How to Respond, Help, your Company Information. All will show up as dedicated accordions on the custom content page. However, if you need additional categories not displayed there (for example, if you have multiple versions of the searches, or if you have multiple companies that should be listed for a particular piece of content), you can define them in additional_context.

additional_context is an array of objects, each of which will be a new accordion. Each object can have up to five fields:

• title: Text-only, provides the name of the accordion. Default if blank: “Additional Context”
• open_panel: Whether the accordion should be open by default or not.
• detail: Supports markdown. An optional text block.
• link: An optional URL for users to learn more. If defined, a button will be added with the text “Learn More…”
• search_label: Text-only, provides the label for the search, directly before the pre. Default if blank: “Search”
• search_lang: Optional definition of which language the below code is in. Can leave empty for normal SPL, but could be provided as conf if you are printing config file options. In the event you really needed to print options such as python, the default languages for highlight.js are supported. (Valid options: properties, python, rust, less, perl, diff, scss, bash, shell, makefile, json, ini, http, coffeescript, css, objectivec, ruby, yaml, java, sql, apache, kotlin, xml, markdown, swift, plaintext, typescript, nginx, go, javascript, php, cs, lua, cpp)
• search: Can present SPL or any other raw code. It is implemented via the following code, and then syntax highlighting is applied. If the lang is SPL (or empty, which defaults to SPL) then an Open In Search button will be added below:

if(additional_context[num].search){
    output += $("<div>").append($("<pre>").attr("class", "search").append($("<code>").attr("class", lang).text(obj.search))).html()
}

Example:

{
    "additional_context": [
        {
            "search": "index=* sourcetype=ponies",
            "open_panel": true,
            "title": "Additional Potential Search",
            "link": "https://www.splunk.com/blog/2016/09/28/buttercup-games.html",
            "detail": "### Background\n* You may wish to look at all data regarding ponies.\n* Here you will find all of the detail produced by Buttercup Labs Pony Monitoring"
        },
        {
            "title": "Conf file for Adding a New SSE Channel",
            "search": "[ButtercupLabs]\nchannel=ButtercupLabs\nname=Buttercup Labs\ndescription=Buttercup Labs produces high quality security analytics run through Splunk Enterprise. While Buttercup sells a commercial threat analytics app, they have also released a variety of community content. All is available through Splunk Security Essentials.\ntype=app\napp_context=Splunk_Security_Essentials\ncontent_download_url=https://go.splunksecurityessentials.com/myContentLocation",
            "detail": "Adding this file and restarting Splunk will register a new content source into SSE.",
            "search_lang": "conf",
            "search_label": "Example Configuration",
            "open_panel": true
        }
    ]
}

The above JSON will produce the following output:

Formatting

Fields

The following fields support Markdown formatting (implemented via showdownjs – https://github.com/showdownjs/showdown/wiki/Showdown’s-Markdown-syntax):

• description (markdown highly discouraged, but available)
• howToImplement
• knownFP
• operationalize
• help
• company_description
• In additional_context: detail

Style Guide

Most markdown here is very user-friendly, but one should generally avoid using # or ## to indicate a header or a second-level header. ### (or a third level header) will be translated into an h3 tag which is as prominent as it generally makes sense.

Verifying Your Configuration

You can always view your content by installing your app on your local system. You can view the details of all your content by running a search for:

| sseanalytics include_json=true | search channel=ButtercupLabs | table id summaries *

Populating Data Inventory

One of the primary configuration interfaces for Splunk Security Essentials is the Data Inventory configuration. By default, custom content will not show there, and you can map your data_source_categories field to standard Data Source Categories (DSCs), e.g., DS001MAIL-ET02Receive – see the SPL at the end of “Field Descriptions” for more examples.

However, if you would like to have your content show up as a Vendor-Specific data source category, add the following to your configuration:

{
    "create_data_inventory": true
}

SSE will then create a custom DSC with the data provided in the company_* fields (see “Company Info” above). It is important that you provide the full company_info for any fields defined as create_data_inventory=true.

Implementation Detail

When pullJSON.py pulls in the data_inventory.json, it will look in the custom_content kvstore collection (the back-end of the partner framework) it will take any detections that have a create_data_inventory=true configuration. For the first piece of content that it finds, it will add a new item to data_inventory output using the following logic:

dscid = "VendorSpecific-" + row['channel']
baseSearch = "index=NOTAPPLICABLE TERM(No baseSearch Provided)"
legacyName = "Unknown Channel: " + row["channel"]
shortUnifiedName = "Unknown Channel: " + row["channel"]
description = "No Description Provided"
commonProductNames = []
if "company_description" in customJSON:
    description = customJSON['company_description']
if "company_name" in customJSON:
    legacyName = customJSON['company_name']
    shortUnifiedName = customJSON['company_name']
    commonProductNames.append(customJSON['company_name'])
if "company_base_spl" in customJSON:
    baseSearch = customJSON['company_base_spl']

data_inventory["VendorSpecific"]["eventtypes"][dscid] = {
    "baseSearch": baseSearch,
    "legacy_name": legacyName,
    "short_unified_name": shortUnifiedName,
    "description": description,
    "name": legacyName,
    "common_product_names": commonProductNames
}

Warnings and Challenges

Generally this process is relatively straightforward, but there are some limitations and challenges.

1. Because this process only ships JSON files into SSE, make sure that you host your company logo online.
2. Try to avoid adding an excessive amount of text in your content. This is both because users rarely enjoy pages of reading when sentences would do, but more so because many pages require downloading the entirety of the content in SSE. This can result in a large download when people view certain pages in the app. While Splunk compresses by default, there were issues seen in the past where this download became very large, creating a bad user experience.
3. Splunk Security Essentials doesn’t support adding demo data, or different versions of searches through this interface. If you have a strong need for this, please reach out to your Splunk team.

Security and Data Cleaning

Overview

All data provided via the partner framework is automatically cleaned to ensure the safety of users. This configuration is hard-coded in generateShowcaseInfo.py, the core REST endpoint that powers SSE.

Fields

For any content pulled from the partner content framework, only the following fields are allowed, with the corresponding data cleaning method provided:

Cleaning Methods

Text

Cleaned in python via BeautifulSoup (bs4) to strip any HTML.

obj[field] = BeautifulSoup(obj[field], "lxml").text

Number

Cleaned in python via BeautifulSoup (bs4) to strip any HTML.

obj[field] = BeautifulSoup(obj[field], "lxml").text

Boolean

Checked to make sure it is a boolean field, and otherwise deleted

if not isinstance(obj[field], bool):
    debug.append({"status": "WARN", "msg": "clean_content, deleting field because it's not actually a bool", "path": path, "field": field, "value": obj[field]})
    del obj[field]

Object

The clean function cascades to the object to evaluate all of the fields. The key checking respects the sequence by appending the path. Note: This is not currently used but is available for future enhancement requests.

Array

Each row in the array should be an object. The code takes each row, and then recursively calls clean_content for that row.

if key_checking[path + field] == "array":
    for i in list(range(0, len(obj[field]) )):
        if isinstance(obj[field][i], object):
            obj[field][i] = clean_content(obj[field][i], key_checking, path="")

SPL

This is the only content for which there is no cleaning that occurs in Python. All code is input into the app via Jquery into a pre tag via .text(), to ensure that there is no code execution.

output += $("<div>").append($("<pre>").text(additional_context[num].search)).html()

Troubleshooting Data Cleaning

As with any scenario involving the troubleshooting ShowcaseInfo, it is best to check the debug output for any logs via the JavaScript console. Whenever fields are deleted from SSE, they will be logged into the debug:

require(['json!' + $C['SPLUNKD_PATH'] + '/services/SSEShowcaseInfo?bust=' + Math.random()], function(showcase){
    window.debug_showcase = showcase; 
    console.log("Got Showcase", showcase.debug); 
})