Hosts with Varied and Future Timestamps
One technique for foiling correlation searches is to alter the system time. This search will detect this scenario.
(and should be fixed)
How to Implement
This search should work universally on all Splunk environments, since it uses Splunk internal fields.
Known False Positives
This should not fire often -- the idea is it should only fire when the system time changes, which should not happen often. The only known false positives are hosts that are sending data with multiple timestamps simultaneously, which is usually an instance of an incorrect _time extraction (which should be fixed) or a data source with known lag (which should be filtered out).
How To Respond
When timestamps are detected as being incorrect, it should be noted which system's timestamps are off and when it started. Notification of the system owner needs to be made to correct this condition, but additional investigation may be required to determine if this was an accidental or malicious configuration change and by whom.
Hosts with Varied and Future Timestamps Help
This example leverages the Simple Search assistant. Here we are looking through all Splunk logs for hosts that begin sending logs at very different timestamps. The goal is to detect an attacker who suddenly sets the year to 2020 or some point in the future to evade detection. We use the tstats command here, because we are only looking at indexed field, and it is super fast for those use cases. If we find a broad range of time for any host, the search surfaces it. We use 21 years into the future for the time range, because that is the maximum time range you can put into a Splunk search. We're looking for a range of more than one hour, just to make sure we're above chance.
SPL for Hosts with Varied and Future Timestamps
Live Data (Auto Accelerated)
|First we use tstats to grab events that were indexed in the last half hour, with timestamps ranging from the last half hour to the distant future.|
|Next we look for the time ranges from a single host. This is a quick and dirty baseline, because if we're constantly running this search and an attacker changes the system time from now to a year from now, there will be some period when there is a massive range of timestamps coming from that host.|
|Finally we filter for those large ranges.|