Microsoft Sysmon Logs

Data Source Onboarding Guide Overview

Overview  

Welcome to the Splunk Data Source Onboarding Guides (DSOGs)!

Splunk has lots of docs, so why are we creating more? The primary goal of the DSOGs is to provide you with a curated, easy-to-digest view of the most common ways that Splunk users ingest data from our most popular sources, including how to configure the systems that will send us data (such as turning on AWS logging or Windows Security's process-launch logs, for example). While these guides won't cover every single possible option for installation or configuration, they will give you the most common, easiest way forward.

How to use these docs: We've broken the docs out into different segments that get linked together. Many of them will be shared across multiple products. We suggest clicking the "Mark Complete" button above to remind yourself of those you've completed. Since this info will be stored locally in your browser, you won't have to worry about it affecting anyone else's view of the document. And when you're reading about ingesting Sysmon logs, for example, it's a convenient way to keep track of the fact that you already installed the forwarder in order to onboard your Windows Security logs.

So, go on and dive right in! And don't forget, Splunk is here to make sure you're successful. Feel free to ask questions of your Sales Engineer or Professional Services Engineer, if you run into trouble. You can also look for answers or post your questions on https://answers.splunk.com/.

General Infrastructure

Instruction Expectations and Scaling  

Expectations

This doc is intended to be an easy guide to onboarding data from Splunk, as opposed to comprehensive set of docs. We've specifically chosen only straightforward technologies to implement here (avoiding ones that have lots of complications), but if at any point you feel like you need more traditional documentation for the deployment or usage of Splunk, Splunk Docs has you covered with over 10,000 pages of docs (let alone other languages!).

Because simpler is almost always better when getting started, we are also not worrying about more complicated capabilities like Search Head Clustering, Indexer Clustering, or anything else of a similar vein. If you do have those requirements, Splunk Docs is a great place to get started, and you can also always avail yourself of Splunk Professional Services so that you don't have to worry about any of the setup.

Scaling

While Splunk scales to hundreds or thousands of indexers with ease, we usually have some pretty serious architecture conversation before ordering tons of hardware. That said, these docs aren't just for lab installs. We've found that they will work just fine with most customers in the 5 GB to 500 GB range, even some larger! Regardless of whether you have a single Splunk box doing everything, or a distributed install with a Search Head and a set of Indexers, you should be able to get the data and the value flowing quickly.

There's one important note: the first request we get for orchestration as customers scale, is to distribute configurations across many different universal forwarders. Imagine that you've just vetted out the Windows Process Launch Logs guide on a few test systems, and it's working great. Now you want to deploy it to 500, or 50,000 other Windows boxes. Well, there are a variety of ways to do this:

  • The standard Splunk answer is to use the Deployment Server. The deployment server is designed for exactly this task, and is free with Splunk. We aren't going to document it here, mostly because it's extremely well documented by our EDU and also docs.splunk.com, here.
  • If you are a decent sized organization, you've probably already got a way to deploy configurations and code, like Puppet, Chef, SCCM, Ansible, etc. All of those tools are used to deploy splunk on a regular basis. Now, you might not want to go down this route if it requires onerous change control, or reliance on other teams, etc. -- many large Splunk environments with well developed software deployment systems prefer to use the Deployment Server because it can be owned by Splunk and is optimized for Splunk's needs. But many customers are very happy with using Puppet to distribute Splunk configurations.
Ultimately, Splunk configurations are almost all just text files, so you can distribute the configurations with our packaged software, with your own favorite tools, or even by just copying configuration files around.

Indexes and Sourcetypes Overview  

Overview

The DSOGs talk a lot about indexes and sourcetypes. Here's a quick overview.

Splexicon (Splunk's Lexicon, a glossary of Splunk-specific terms) defines an index as the repository for data in Splunk Enterprise. When Splunk Enterprise indexes raw event data, it transforms the data into searchable events. Indexes are the collections of flat files on the Splunk Enterprise instance. That instance is known as an Indexer because it stores data. Splunk instances that users log into and run searches from are known as Search Heads. When you have a single instance, it takes on both the search head and indexer roles.

"Sourcetype" is defined as a default field that identifies the data structure of an event. A sourcetype determines how Splunk Enterprise formats the data during the indexing process. Example sourcetypes include access_combined and cisco_syslog.

In other words, an index is where we store data, and the sourcetype is a label given to similar types of data. All Windows Security Logs will have a sourcetype of WinEventLog:Security, which means you can always search for source=*wineventlog:security (when searching, the word sourcetype is case sensitive, the value is not).

Why is this important? We're going to guide you to use indexes that our professional services organization recommends to customers as an effective starting point. Using standardized sourcetypes (those shared by other customers) makes it much easier to use Splunk and avoid headaches down the road. Splunk will allow you to use any sourcetype you can imagine, which is great for custom log sources, but for common log sources, life is easier sticking with standard sourcetypes. These docs will walk you through standard sourcetypes.

Implementation

Below is a sample indexes.conf that will prepare you for all of the data sources we use in these docs. You will note that we separate OS logs from Network logs and Security logs from Application logs. The idea here is to separate them for performance reasons, but also for isolation purposes-you may want to expose the application or system logs to people who shouldn't view security logs. Putting them in separate indexes prevents that.

To install this configuration, you should download the app below and put it in the apps directory.

For Windows systems, this will typically be: c:\Program Files\Splunk\etc\apps. Once you've extracted the app there, you can restart Splunk via the Services Control Panel applet, or by running "c:\Program Files\Splunk\bin\splunk.exe" restart.

For Linux systems, this will typically be /opt/splunk/etc/apps/. Once you've extracted the app there, you can restart Splunk by running /opt/splunk/bin/splunk restart.

You can view the indexes.conf below, but it's easiest to just click Click here to download a Splunk app with this indexes.conf, below.

Splunk Cloud Customers: You won't copy the files onto your Splunk servers because you don't have access. You could go one-by-one through the UI and create all of the indexes below, but it might be easiest if you download the app, and open a ticket with CloudOps to have it installed.


Sample indexes.conf
# Overview. Below you will find the basic indexes.conf settings for
# setting up your indexes in Splunk. We separate into different indexes 
# to allow for performance (in some cases) or data isolation in others. 
# All indexes come preconfigured with a relatively short retention period 
# that should work for everyone, but if you have more disk space, we 
# encourage (and usually see) longer retention periods, particularly 
# for security customers.

# Endpoint Indexes used for Splunk Security Essentials. 
# If you have the sources, other standard indexes we recommend include:
# epproxy - Local Proxy Activity

[epav]
coldPath = $SPLUNK_DB/epav/colddb
homePath = $SPLUNK_DB/epav/db
thawedPath = $SPLUNK_DB/epav/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[epfw]
coldPath = $SPLUNK_DB/epnet/colddb
homePath = $SPLUNK_DB/epnet/db
thawedPath = $SPLUNK_DB/epnet/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[ephids]
coldPath = $SPLUNK_DB/epmon/colddb
homePath = $SPLUNK_DB/epmon/db
thawedPath = $SPLUNK_DB/epmon/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[epintel]
coldPath = $SPLUNK_DB/epweb/colddb
homePath = $SPLUNK_DB/epweb/db
thawedPath = $SPLUNK_DB/epweb/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[oswin]
coldPath = $SPLUNK_DB/oswin/colddb
homePath = $SPLUNK_DB/oswin/db
thawedPath = $SPLUNK_DB/oswin/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[oswinsec]
coldPath = $SPLUNK_DB/oswinsec/colddb
homePath = $SPLUNK_DB/oswinsec/db
thawedPath = $SPLUNK_DB/oswinsec/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[oswinscript]
coldPath = $SPLUNK_DB/oswinscript/colddb
homePath = $SPLUNK_DB/oswinscript/db
thawedPath = $SPLUNK_DB/oswinscript/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[oswinperf]
coldPath = $SPLUNK_DB/oswinperf/colddb
homePath = $SPLUNK_DB/oswinperf/db
thawedPath = $SPLUNK_DB/oswinperf/thaweddb
frozenTimePeriodInSecs = 604800 
#7 days

[osnix]
coldPath = $SPLUNK_DB/osnix/colddb
homePath = $SPLUNK_DB/osnix/db
thawedPath = $SPLUNK_DB/osnix/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[osnixsec]
coldPath = $SPLUNK_DB/osnixsec/colddb
homePath = $SPLUNK_DB/osnixsec/db
thawedPath = $SPLUNK_DB/osnixsec/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[osnixscript]
coldPath = $SPLUNK_DB/osnixscript/colddb
homePath = $SPLUNK_DB/osnixscript/db
thawedPath = $SPLUNK_DB/osnixscript/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[osnixperf]
coldPath = $SPLUNK_DB/osnixperf/colddb
homePath = $SPLUNK_DB/osnixperf/db
thawedPath = $SPLUNK_DB/osnixperf/thaweddb
frozenTimePeriodInSecs = 604800 
#7 days

# Network Indexes used for Splunk Security Essentials
# If you have the sources, other standard indexes we recommend include:
# netauth - for network authentication sources
# netflow - for netflow data
# netids - for dedicated IPS environments
# netipam - for IPAM systems
# netnlb - for non-web server load balancer data (e.g., DNS, SMTP, SIP, etc.)
# netops - for general network system data (such as Cisco iOS non-netflow logs)
# netvuln - for Network Vulnerability Data

[netdns]
coldPath = $SPLUNK_DB/netdns/colddb
homePath = $SPLUNK_DB/netdns/db
thawedPath = $SPLUNK_DB/netdns/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[mail]
coldPath = $SPLUNK_DB/mail/colddb
homePath = $SPLUNK_DB/mail/db
thawedPath = $SPLUNK_DB/mail/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[netfw]
coldPath = $SPLUNK_DB/netfw/colddb
homePath = $SPLUNK_DB/netfw/db
thawedPath = $SPLUNK_DB/netfw/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[netops]
coldPath = $SPLUNK_DB/netops/colddb
homePath = $SPLUNK_DB/netops/db
thawedPath = $SPLUNK_DB/netops/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[netproxy]
coldPath = $SPLUNK_DB/netproxy/colddb
homePath = $SPLUNK_DB/netproxy/db
thawedPath = $SPLUNK_DB/netproxy/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days

[netvpn]
coldPath = $SPLUNK_DB/netvpn/colddb
homePath = $SPLUNK_DB/netvpn/db
thawedPath = $SPLUNK_DB/netvpn/thaweddb
frozenTimePeriodInSecs = 2592000
#30 days


# Splunk Security Essentials doesn't have examples of Application Security, 
# but if you want to ingest those logs, here are the recommended indexes:
# appwebint - Internal WebApp Access Logs
# appwebext - External WebApp Access Logs
# appwebintrp - Internal-facing Web App Load Balancers
# appwebextrp - External-facing Web App Load Balancers
# appwebcdn - CDN logs for your website
# appdbserver - Database Servers
# appmsgserver - Messaging Servers
# appint - App Servers for internal-facing apps 
# appext - App Servers for external-facing apps 

Validation

Once this is complete, you will be able to find the list of indexes that the system is aware of by logging into Splunk, and going into Settings -> Indexes.

Forwarder on Windows Systems  

Overview

Installing the Windows forwarder is a straightforward process, similar to installing any Windows program. These instructions will walk you through a manual instruction for getting started (perfect for a lab, a few laptops, or when you're just getting started on domain controllers).

Implementation

Note for larger environments: When you want to automatically roll out the forwarder to hundreds (or thousands or hundreds of thousands) of systems, you will want to leverage your traditional software-deployment techniques. The Splunk forwarder is an MSI package and we have docs on recommended ways to deploy it:

Of course, you can also deploy it with traditional system-configuration management software. This can vary a lot from environment to environment. For this doc we'll just walk you through the installation so that you know what's coming.

The first thing to do is download the Universal Forwarder from Splunk's website (https://www.splunk.com/en_us/download/universal-forwarder.html). This is a separate download from the main Splunk installer, as the universal forwarder is lightweight, so it can be installed on all of the systems in your environment. Most users today will download the x64 version as an MSI installer.

When you double click the downloaded file, the standard MSI installer will appear.

The initial installer screen for the Splunk Forwarder. Click Next to continue, don't worry about customizing the options. You can also install the package silently.

Don't worry about the Cloud checkbox -- we will use the same settings for both.

While you can click "Customize Settings" here and manually insert the address of your indexers or manually choose the log sources you would like to index, etc., we generally don't recommend that, unless you're never going to move beyond the one source you're looking at. (Harder to go find those settings and then apply them to other systems.) Ignore "Customize Options" and click on "Next." The setup will now go through its process, and you'll be finished with a freshly installed forwarder. There are three more steps you'll want to take before you can see the data in Splunk:

  • You will need an outputs.conf to tell the forwarder where to send data (next section)
  • You will need an inputs.conf to tell the forwarder what data to send (below, in the Splunk Configuration for Data Source)
  • You will need an indexes.conf on the indexers to tell them where to put the data that is received (Previous section)

Validation

You can now check Task Manager and you should see Splunk running. Alternatively, check under Services in the Control Panel. You will see Splunk listed and started.

Sending Data from Forwarders to Indexers  

Overview

For any Splunk system in the environment, whether it's a Universal Forwarder on a Windows host, a Linux Heavy-Weight Forwarder pulling the more difficult AWS logs, or even a dedicated Search Head that dispatches searches to your indexers, every system in the environment that is not an indexers (i.e., any system that doesn't store its data locally) should have an outputs.conf that points to your indexers.

Implementation

Fortunately the outputs.conf will be the same across the entire environment, and is fairly simple. There are three steps:

  1. Create the app using the button below (SplunkCloud customers: use the app you received from SplunkCloud).
  2. Extract the file (it will download a zip file).
  3. Place in the etc/apps directory.

For Windows systems, this will typically be: c:\Program Files\Splunk\etc\apps. Once you've extracted the app there, you can restart Splunk via the Services Control Panel applet, or by running "c:\Program Files\Splunk\bin\splunk.exe" restart.

For Linux systems, this will typically be /opt/splunkforwarder/etc/apps/. Once you've extracted the app there, you can restart Splunk by running /opt/splunk/bin/splunk restart.

For customers not using SplunkCloud:

Sample outputs.conf
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = MySplunkServer.mycompany.local:9997

[tcpout-server://MySplunkServer.mycompany.local:9997]

Here is the completed folder.

Validation

Run a search in the Splunk environment for the host you've installed the forwarder on. E.g., index=* host=mywinsystem1*

You can also review all hosts that are sending data from via | metadata index=* type=hosts

Splunk Configuration for Data Source

Sysmon Overview  

Sysmon, a component of Microsoft’s Sysinternals suite of Windows utilities, is a very powerful host-level tool that can assist you in detecting advanced threats on your network by providing intricate host-operation details in real time. In contrast to common Antivirus/Host-Based Intrusion-detection (HIDS) solutions, Sysmon performs system activity deep monitoring and logs high-confidence indicators of advanced attacks. Read more about Sysmon in Microsoft’s documentation.

A fantastic way to collect detailed information about your Windows endpoints in Splunk, Sysmon is free of charge, installs painlessly on many variants of Windows, and integrates well with Splunk deployments. In fact, Mark Russinovich, Sysmon’s author, has spoken about Sysmon at the past two RSA conferences and showcases Splunk as an excellent mechanism for the collection and analysis of Sysmon data. Check out his slide deck.

As of the most recent version, Sysmon is capable of producing extensive details on the following一much of it very useful in the early detection of malicious code execution or other nefarious behavior:

  • Process executions, including parent/child relationships, user that launched process, and hash data
  • File creations
  • File creation time changes
  • Network activity, down to the process level
  • Image loads
  • Creation of  remote threads
  • Interprocess accesses
  • Windows registry modifications
  • NTFS alternate data stream (ADS) creations
  • Pipe creations and connections
  • WMI event monitoring

Assuming you have already installed a recent version of the Splunk universal forwarder (UF) for Windows, follow these high-level steps to configure Splunk to properly ingest Sysmon data:

  1. Install Sysmon on the endpoint running the UF and configure it to collect the information you require.
  2. Configure Splunk to properly parse the Sysmon event logs.
  3. Configure Splunk to collect the Sysmon event log from the endpoint.

Sysmon Sizing Estimate  

Sizing can vary significantly, depending on the Sysmon configuration, but the SwiftOnSecurity configuration linked below reduces the information returned by Sysmon to the minimum needed to be useful. Since the UF operation can also result in significant process-execution events, we also recommend specific configurations to filter out this data here. We will cover this in the Installing Sysmon section below.

A properly configured Windows endpoint running Sysmon will result in 2-4MB of Sysmon data ingested in Splunk daily—sometimes much less. Of course, particularly busy or compromised machines may generate more data. You may also select “critical” machines or machines owned by “most likely targets” and voluntarily increase the verbosity of logging on these systems. For example, the NetworkConnect (Event Code 3) and Image Load (Event Code 7) logging may be increased for these systems.

For a recent success story involving the deployment of Sysmon (plus other Windows endpoint data sources) that resulted in 10-12MB of data per endpoint ingested, review the .conf2017 presentation delivered by Splunk customer TransAlta.

Install the Technology Add-On -- TA  

The Sysmon TA is a Splunk app that configures Splunk to understand the Sysmon data format. It is located on SplunkBase.

Splunk recommends that you install TAs (particularly the files props.conf and transforms.conf) on all Splunk installs, including the universal forwarder. This isn’t  necessarily required for Sysmon, but we recommend you stick with the best practices across the board.

Installing on Search Heads, Indexers, or on a Single Splunk Environment

While you can opt to install the TA by copying the files into your directory structure in the same way explained for forwarders below, you can take a simpler approach for a Splunk system with UI. In fact, it’s the same approach you probably took for installing Splunk Security Essentials in the first place—via the web interface, as documented here. The Sysmon TA does not have a user interface, so it will not appear in the list of apps. However, once it’s installed, you will be able to see it under Manage Apps.

Installing on forwarders

  1. To install the app, start by downloading the file from the SplunkBase, then extract it. Note: The app itself is a .tgz file, or a gzipped tarball. If you’re running a pure Windows environment, this means that you will need a third-party program to extract it. Fortunately, .tgz is the most common format in the world after .zip, so virtually any extraction program you have (WinZip, 7z, WinRAR, etc.) will all extract it.

  2. Once you have the extracted folder, move it into

    %SPLUNK_HOME%/etc/apps/
    folder. For most modern Splunk environments, that will be
    C:\Program Files\SplunkUniversalForwarder\etc\apps
    .

  3. Once you’ve extracted the app, you must restart Splunk, either via the Services Control Panel applet or by running

    C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe restart
    .

Validation

You can make sure that Splunk has picked up the presence of the app by running:

C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe
display app, will, after asking you to log in, provide you with a list of installed apps. Usually, however, if you see the folder listed alongside the other apps (learned, search, splunk_httpinput, etc.), you can be assured that it’s there.

Splunk Cloud Customers: you won't be copying any files or folders to your indexers or search heads, but good news! Even though the Sysmon TA is not Cloud Self-Service Enabled, you will still be able to open a ticket with Cloud Ops and be ready to go in short order.

General Windows Indexes and Sourcetypes  

Overview

Amongst Splunk’s 15000+ customers, we’ve done a lot of implementations, and we’ve learned a few things along the way. While you can use any sourcetypes or indexes that you want in the land of Splunk, we’ve found that the most successful customers follow specific patterns, as it sets them up for success moving forward.

Implementation

The most common Windows data types are the Security Log, System Log, and Application Log, but there are a few others as well including Microsoft Sysmon. Here are our most commonly used Windows Data Types and the recommended indexes and sourcetypes.

Data TypeInput (inputs.conf, below)SourceIndexNotes
Windows Security LogsWinEventLog://Securitywineventlog:securityoswinsecWe leverage a blacklist for common “noise” events, below.
Windows Application LogsWinEventLog://Applicationwineventlog:applicationoswin
Windows System LogsWinEventLog://Systemwineventlog:systemoswin
Windows Update Logmonitor://$WINDIR\WindowsUpdate.logWindowsUpdateLogoswinsec
Microsoft Sysmon LogsWinEventLog://Microsoft-Windows-Sysmon/OperationalXmlWinEventLog:Microsoft-Windows-Sysmon/OperationalepintelBased on the sysmon sysinternals tool, not out of the box.

If you have already started ingesting the data sources into another index, then you can usually proceed (though consider if you should separate Windows Security logs from Process Launch Logs and both from Application and System logs, based on who likely will need access or be prohibited access). If you have already started ingesting data with a different sourcetype, we would recommend you switch over to the standardized sourcetypes if at all possible. If you're not using the Splunk TA for Windows to ingest data, then keep in mind you may need to go through extra work to align field names to get value out of Splunk Security Essentials, and other Splunk content.

To support your Windows sources, follow the procedure mentioned above in General Infrastructure - Indexes and Sourcetypes to add the new indexes for the data you will be bringing in (generally it’s easiest if you just create oswin, oswinsec, epintel).

For the sourcetypes and monitor statements, we will show those next in the Configuration Files.

Configuration Files  

Overview

Configuration files for Sysmon are pretty simple. In this case, we just have a single inputs.conf file that will go on the Windows hosts you will be monitoring. You’ll need to collect another Windows Event Log to collect the information from Sysmon running on the endpoint. While Splunk supports a few mechanisms for collecting event logs, almost all customers follow our Professional Services’ best practices and pull them in via the universal forwarder (UF).

As detailed in Instruction Expectations and Scaling, you will need some mechanism to distribute these files to the hosts you’re monitoring. For initial tests or deployments to just your most sensitive systems, it is easy to copy the files to the hosts. For larger distributions, you can use the Splunk deployment server or use another code-distribution system, such as SCCM, Puppet, Chef, Ansible, or others.

Implementation

Distribute the below inputs.conf file to your hosts in the %SPLUNK_HOME%\etc\apps\TA-microsoft-sysmon\local folder. If the folder doesn’t exist, you will need to create it. For most customers, the path to this file will end up being C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-microsoft-sysmon\local\inputs.conf.


inputs.conf(Download File)
[WinEventLog://Microsoft-Windows-Sysmon/Operational]
disabled = false
renderXml = 1
index = epintel

Verification

If all is properly configured, you should be able to find your data using the following Splunk search:

index=* source=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational

You should see data coming into Splunk and then verify correct timestamps, event breaking, and field extraction.

System Configuration

Single System Sysmon Installation  

Implementation

Download Sysmon.

Sysmon can generate large amounts of logs. You will want to tune your collection configuration. We recommend starting with this Sysmon configuration: https://github.com/SwiftOnSecurity/sysmon-config. Advanced users may also want to look at https://github.com/MHaggis/sysmon-dfir.

In either case, be sure to include configuration lines to exclude Splunk forwarder execution activity, especially for process-creation events.

To install on your endpoint, run the following with administrator rights:

sysmon.exe -accepteula -i sysmonconfig-export.xml

Update existing configuration, again, with administrator rights:

sysmon.exe -c sysmonconfig-export.xml

Upon installation, Sysmon will begin logging events to the operational event log.

C:\Windows\System32\winevt\Logs\Microsoft-Windows-Sysmon%4Operational.evtx

Verify your installation

After successful installation, you should be able to see the Sysmon log in your Windows Event Viewer.

(Optional) Large Scale Sysmon Installation  

Large-scale deployments of Sysmon can be automated using Group Policy Objects (GPO), as described here.