Splunk: troubleshooting missing, late and mis-parsed data

Skills you'll practise

• Work through Splunk's 'I can't find my data' checks: index, role permissions, time range, forwarders, search heads, licence and the query
• Prove whether a forwarder is connected using _internal metrics and splunkd.log, and fix receiving-port mistakes
• Use btool with --debug to find which file sets a configuration value, and know its limitation
• Explain timestamp and time zone precedence and where those props.conf settings must live
• Measure indexing delay with _indextime and match the pattern to a cause
• Describe what happens to indexing and searching during a licence violation

Course outline

1. 1.Splunk: finding missing and late dataVideo · 2 min
2. 2.'I can't find my data': the first checksLesson · 16 min
3. 3.Forwarders, receivers, queues and btoolLesson · 17 min
4. 4.Timestamps, time zones, line breaking, delay and licenceLesson · 17 min
5. 5.Splunk: troubleshooting missing, late and mis-parsed data: knowledge checkKnowledge check · 17 questions
6. 6.Final exam9 questions · passing it completes the course, so people who already know the material can test out

Sources it draws on

The lessons and questions are written from these references, so learners can go back to the original.

• Splunk Enterprise Troubleshooting Manual: I can't find my data!
• Splunk Enterprise Troubleshooting Manual: Use btool to troubleshoot configurations
• Splunk Enterprise Troubleshooting Manual: What Splunk software logs about itself
• Splunk Enterprise Troubleshooting Manual: Event indexing delay
• Splunk Enterprise Forwarding Data: Troubleshoot forwarder/receiver connection
• Splunk Universal Forwarder Manual: Troubleshoot the universal forwarder
• Splunk Enterprise Getting Data In: How timestamp assignment works
• Splunk Enterprise Getting Data In: Specify time zones for timestamps
• Splunk Enterprise Getting Data In: Configure event line breaking
• Splunk Enterprise Admin Manual: About license violations