Splunk is one of the most powerful tools a network team can have, and one of the most underused. Many teams send syslog to it and then search for a hostname when something breaks. With a handful of well-built searches, the same data becomes an early-warning system.
Below are ten searches we use repeatedly in data-center and enterprise networks. The examples use Cisco IOS-style syslog messages because they are common and easy to read. The same patterns work for Juniper, Arista and others with different message text.
Assumptions: network syslog is indexed in
index=network. If your devices have a vendor add-on from Splunkbase installed, many fields are extracted automatically. Here we userexso the searches work either way.
1. Which devices are the noisiest?
index=network earliest=-24h
| stats count by host
| sort - count
| head 20
Start every noise-reduction effort here. A device sending ten times more logs than its peers usually has a real problem, or a logging configuration that needs fixing.
2. Break down messages by severity
Cisco messages follow the pattern %FACILITY-SEVERITY-MNEMONIC. Extract all three:
index=network
| rex "%(?<facility>[A-Z0-9_]+)-(?<severity>[0-7])-(?<mnemonic>[A-Z0-9_]+)"
| timechart span=1h count by severity
Severity 0–2 (emergency, alert, critical) should be rare. A rising line there deserves attention even if nobody has complained yet.
3. Interfaces going down
index=network "%LINEPROTO-5-UPDOWN" "changed state to down"
| rex "Interface (?<interface>[^,]+), changed state to (?<state>\w+)"
| stats count latest(_time) as last_down by host interface
| convert ctime(last_down)
| sort - count
This gives you a ranked list of the least stable links, a great input for your weekly review.
4. Detect flapping interfaces
A flap is several transitions in a short period. Bucket events into 5-minute windows:
index=network "%LINEPROTO-5-UPDOWN"
| rex "Interface (?<interface>[^,]+), changed state to (?<state>\w+)"
| bin _time span=5m
| stats count by _time host interface
| where count >= 6
Six or more transitions in 5 minutes means the link went up and down at least three times. Save this as an alert with throttling on host,interface so you get one notification per flapping link, not one per run.
5. BGP neighbor changes
index=network "%BGP-5-ADJCHANGE"
| rex "neighbor (?<peer>\S+)(?: vpn vrf \S+)? (?<state>Up|Down)"
| stats count(eval(state="Down")) as downs count(eval(state="Up")) as ups latest(state) as current_state by host peer
| where downs > 0
| sort - downs
current_state tells you at a glance which sessions are still down right now.
6. Who changed the configuration?
index=network "%SYS-5-CONFIG_I"
| rex "Configured from (?<method>\S+) by (?<user>\S+)"
| table _time host user method
| sort - _time
Compare the output with your approved change records. Configuration changes outside a change window are worth a conversation, and sometimes a security review.
7. Failed logins to network devices
index=network "%SEC_LOGIN-4-LOGIN_FAILED"
| rex "\[user: (?<user>[^\]]*)\] \[Source: (?<src>[^\]]+)\]"
| stats count dc(host) as devices_targeted values(user) as users by src
| where count > 10
| sort - count
A single source failing against many devices with many usernames is a classic sign of password spraying. It is exactly the kind of signal a network team can hand to the security team.
8. Devices that have gone silent
Missing logs can be as important as bad logs. This search finds devices that have not sent anything in the last hour:
| tstats latest(_time) as last_seen where index=network earliest=-7d by host
| eval minutes_silent = round((now() - last_seen) / 60)
| where minutes_silent > 60
| convert ctime(last_seen)
| sort - minutes_silent
tstats works on indexed metadata, so it is fast even across large volumes. Silent devices might be down, misconfigured or pointing at the wrong syslog server.
9. Changes followed by problems
Correlate a configuration change with a routing adjacency loss on the same device within 15 minutes:
index=network ("%SYS-5-CONFIG_I" OR "%BGP-5-ADJCHANGE" OR "%OSPF-5-ADJCHG")
| transaction host maxspan=15m startswith="CONFIG_I" endswith=eval(match(_raw, "Down|FULL to DOWN"))
| table _time host duration eventcount
transaction is resource-intensive, so run this over short time ranges or as a scheduled search. The results are a strong starting point for post-incident reviews.
10. Rare messages
Common messages are usually harmless. Rare ones are often interesting:
index=network earliest=-7d
| rex "%(?<facility>[A-Z0-9_]+)-(?<severity>[0-7])-(?<mnemonic>[A-Z0-9_]+)"
| rare limit=25 facility mnemonic
Run this weekly. It regularly surfaces hardware warnings, parity errors and software bugs that never made it into anyone's alarm rules.
Turning searches into alerts
A search becomes useful operationally when it runs on its own. For each alert:
- Schedule it at an interval that matches urgency, for example every 5 minutes for flaps and hourly for silent devices.
- Throttle by the fields that identify the object (
host,interface,peer) to avoid repeat notifications. - Route it to the right place: a ticketing system, a chat channel or your event manager. Send it to the same event pipeline as your SNMP alarms so operators have one console.
- Document it. Each alert should have a short description of what it means and the first troubleshooting steps.
Performance tips
- Always specify
index=and a time range. Unbounded searches are slow for everyone. - Filter early. Put literal strings like
"%BGP-5-ADJCHANGE"in the base search, not in a laterwhere. - Prefer
statsovertransactionwhen you can. - For dashboards that many people open, use scheduled searches or summary indexing instead of running heavy searches on every page load.
Key takeaways
- A few well-built searches turn syslog from an archive into an early-warning system.
- Look for instability (flaps, BGP changes), risk (config changes, failed logins) and gaps (silent devices).
- Use rare-event searches to find problems nobody has written rules for yet.
- Promote the best searches to alerts, throttle them, and feed them into the same pipeline as your other alarms.