This is a writeup of notes from Splunk’s “Intro To Splunk” E-Learning course. At the end of this course I took a quiz and earned a certificate!

Intro to Splunk

Splunk is a unified data platform that allows teams to work together or individually, to ensure critical systems stay secure and reliable.

Splunk’s heart is the index, which contains data from servers, network devices, and web apps. The indexer is a factory, the data is raw materials. As raw material enters the factory, it goes to inspectors.

The inspectors label the data with a source type, and then the source type is used to break the data into single events.

Once data is ingested to the index, it’s available for searching and analyzing. You can enter a query into the Splunk search bar, you can find events that contain values across multiple data sources. You can then analyze and run statistics on events using the Splunk Search Language (SPL).

Web Interface

The Splunk interface contains multiple different configurable apps that can be thought of as different workspaces. The apps you can see and use are defined by a Splunk administrator.

There are three default roles in Splunk enterprise:

  • Administrator
    • Installs apps
    • Ingests data
    • Creates knowledge objects for all users
  • Power
    • Create and share knowledge objects with all users of an app
    • Perform real-time searches
  • User
    • Can only see own knowledge objects and those that have been shared with them

The home app in Splunk is a quick way to explore the enterprise system. You can launch and manage other Splunk apps, find documentation, and set a custom dashboard as your default. Admins can also add apps and data from here.

Hundreds of apps are available via Splunkbase, and you can even develop your own.

The search & reporting app provides a default interface for searching and analyzing data. There are 8 main components to this app’s interface.

  • The Splunk Bar
    • Appears on every Splunk page, can be used to switch between apps, edit your account, view system level messages, manage and edit Splunk configuration, monitor search jobs, and find help
  • The App Bar
    • Allows you to navigate application
  • Search Bar
    • Allows you to run searches
  • Time Range Picker
    • Retrieve events over a specific search period
  • Data Summary
    • Gives you a breakdown of data indexed by host, source, and sourcetypes
  • Table Views is a UI driven way to explore and prepare your data without the need for using SPL.
  • Search History
    • Allows you to view and re-run past searches. The search history can be narrowed by entering search terms into the filter window.

To see if someone is failing to authenticate to a server monitored by Splunk, you can type failed into the searchbar. The Splunk Search Assistant will display matching terms and contextual matches. We can select a period of time in which to view these failed authentications in the time range picker.

Limiting a search by time is key to faster results and is a best practice for every search!

Once the time range is selected, the search is sent to Splunk where it becomes a search job.

You can save the search as a knowledge object.

The events tab displays events returned from the search, and the fields extracted from the events.

The patterns tab allows you to see patterns in the data, allowing you to get a better understanding of what’s happening in your data.

If the data is generating visuals, it’ll be available in the statistics or visualization tabs.

  • Commands that create statistics and visualizations are called transforming commands

Right above the search results tab, we can see the total events and an option to return a random sample of the events.

The search action buttons allow you to edit, send to background, and delete search jobs. You can also pause, stop, share, print, or export a search job.

By default, a search job remains active for 10 minutes after it is run. After 10 minutes, Splunk runs it again. A shared search job remains active for 7 days.

You can export a search job as raw, CSV, XML, or JSON format.

There are three search modes you can select:

  • Fast
    • Cuts down on field information, field discovery disabled in this mode. It only returns information in default fields and fields required to fulfill the search.
  • Verbose
    • Returns as much field and event data as possible, returning as many fields as it can
  • Smart
    • Default mode, toggles behavior based on type of search being run

The Timeline is a visual representation of events over time.

Exploring Events

When a search is run, it returns a list of events. We can use returned events to expand our searches and interact with our data.

The text we search for is highlighted, and events are returned in reverse chronological order (newest first).

You can roll over text in an event to highlight it, and clicking on it allows you to add that text to the search, exclude events that contain that text from the search, or launch a new search using the text.

For example, as we were searching for failed authentication attempts we can find a log entry with specifically “failed password”. We can click “password” and add that to our search, so now we will only see logs for failed password authentications. If you click the text again, you can remove it from the search.

Selecting the “info” button next to an event allows us to see all the extracted fields for the event.

Using Search Terms

If you want to cast a wide net for any text that might indicate a failure, you can use a wildcard. Adding an asterisk to the end of the text “fail” will return any text that starts with the word fail

ex. fail* will return fail, failure, failed Note that search terms are NOT case sensitive

Upper case Booleans of AND OR and NOT can be used with multiple terms.

ex. If we want to see events with the text “failed” but not “password” we can query failed NOT password Note that if no Boolean is used, and is implied

Boolean operations have an order of evaluation:

  1. NOT
  2. OR
  3. AND Parentheses can be used to control the order of evaluation.

ex. failed NOT (success OR accepted) Splunk first evaluates terms in parentheses

Exact terms can be searched by placing the term in quotes. If you’re searching for terms that contain quotes, you can use a backslash to escape quotes.

Using Commands

SPL is built from 5 components:

  • Search Terms
    • Foundation of queries
  • Commands
    • Tells Splunk what we want to do with the results (creating charts, computing statistics, formatting)
  • Functions
    • Explain how we want to chart, compute, and evaluate results
  • Arguments
    • Variables that we want to apply to the function
  • Clauses
    • Explain how we want results grouped or defined
index=network sourcetype=cisco_wsa_squid usage=Violation | stats count(usage) as Visits

Before the pipe is the search terms, and then we have: stats - Command count - Function (usage) - Argument as - Clause

The query searches our web security appliance data in the network index to count the number of visits to prohibited sites from our environment over the last 30 days. It stores that count in a field called “visits”.

You can add a by clause with cs_username at the end of the query to split the count into individual employees.

You can add a search command to the query which allows you to filter results further down the search pipeline. We can use this to find employees that have violated the web policy more than once!

index=network sourcetype=cisco_wsa_squid usage=Violation | stats count(usage) as Visits by cs_username | search Visits >1

Best Practices in Search Terms

If a command references a specific value, it will be case sensitive.

Using time to limit events returned is the most efficient way to filter events. The less data you have to search, the faster using Splunk will be.

After time - index, source, host, and sourcetype are the most powerful fields.

The more you tell the search engine, the better your results. It’s better to search for “failed password” instead of just “password”.

Inclusion is better than exclusion.

When possible use the OR or IN operators instead of wildcards.

Apply filtering commands as early as possible in your search.

Knowledge Objects

Knowledge Objects are tools help you and your users discover and analyze your data.

Five Categories:

  1. Data Interpretation
  2. Data Classifcication
  3. Data Enrichment
  4. Data Normalization
  5. Data Models

Knowledge Objects can be created by one user and shared with others. They can be saved and reused by multiple people and in multiple apps. They can even be used in search!

Knowledge Managers oversee knowledge object creation. They implement best practices for naming conventions, normalize event data, and create data models for event users.

Creating Reports

Splunk makes saving and sharing searches simple with reports.

Once you’ve prepared a search query that shows the data you want to share, you can go to save as and then “Report”.

You can title the report, give a description, confirm how you want to display the data, and whether or not to display the time range picker.

When a report is run, a fresh set of results is returned.

You can access all of your reports from the “reports” tab.

Creating Dashboards

Splunk allows you to visualize data in many ways. Any search that returns statistical data can be viewed as a chart.

We can build dashboards with SPL queries to quickly and easily view common security events. By saving a dashboard, you won’t have to create a new query every time you want to search a specific event.

As you build a dashboard, you can add more and more panels to it to make it verbose.