splunk - ppawelcz/ppwiki GitHub Wiki
Indexer cluster
The manager node manages the cluster. It coordinates the replicating activities of the peer nodes and tells the search head where to find data. It also helps manage the configuration of peer nodes and orchestrates remedial activities if a peer goes offline.
Peer nodes perform the indexing function for the cluster. They receive and index incoming data. They also send replicated data to other peer nodes in the cluster and receive replicated data from other peers
The search head manages searches across the set of peer nodes. It distributes search queries to the peers and consolidates the results
Forwarders consume data from external sources and then forward that data to indexers, which, in clusters, are the peer nodes.
Replication factor(https://docs.splunk.com/Splexicon:Replicationfactor). This determines the number of copies of data the cluster maintains and therefore, the cluster's fundamental level of failure tolerance.
Search factor(https://docs.splunk.com/Splexicon:Searchfactor). This determines the number of searchable copies of data the cluster maintains, and therefore how quickly the cluster can recover its searching capability after a peer node goes down.
Buckets. Buckets are the basic units of index storage. A cluster maintains replication factor number of copies of each bucket.
Fundamentals 1
Field names are case sensitive, values NOT
Status !=200 (events where fields exists)
NOT status 200 (with events where fields does not exists)
top limit=5
rare
stats command - calculates statistics on data matches search criteria
functions: count, dc, sum, avg, list-list all values of a given field, values-list unique values
Lookups pull data from standalone files at search time and add it to search results
Command inputlookup is to load results from specified static lookup
| inputlookup products.csv
Command lookup
If OUTPUT not specified - return all fields
If OUTPUT fileds overwrite existing fields
OUTPUTNEW when you do not want to overwrite
outputlookup - Writes search results to a static lookup table, or KV store collection
Splunk Fundamentals 2
Transforming commands - transform event into data table
stats, chart over by, timechart
timewrap - line compare ex. Week to week
Iplocation - gives longitude, latidude of ip address
geostats - compute stats and render map
addtotals - compute sum of rows and add last row, compute sum of columns and add last column
Search, where
Search - field values case insensitive
Where - filed values case sensitive, two values
Where like
transaction - command create single event from group of events by JSESSIONID
System administration
File precedence (index time/global context) - system local, app local, app default, system default
File precedence (search time) - user local, app local/default, other app local/default, system local/default
splunk show config - in-memory configs
splunk btool - on disk configs
coalesce function returns the first non-null value among its arguments
Indexes:
_internal - splunk logs and metrics
_audit - store audit information
_introspection - track system performance, splunk resource usage
_thefishbucket - contain checkpoint information for files
Indexes stores data in buckets
Hot, warm, cold, frozen
Metrics - set of measures containg timestamp, metric_name, value, dimension
Fishbucker - pointer to where splunk last read the files
Multi search heads - do not share knowledge objects
Clustered search heads - share knowledge objects
Data Administration
splunk show deploy-client - check deployment servers settings
splunk show forward-server - check indexer destination settings
Log, scripted inputs
Windows - evenlogs, WMI, powershell
HEC - token based http input which is secure. Send events to splunk without use of forwarders
transforms.conf, props.conf - mask/delete row data, overwrite sourcetype/host, prevent unwanted events to be indexed
SEDCMD - modify raw data. Only used to mask or truncate data
SH Cluster
A search head cluster is a group of search heads that work together to provide scalability and high availability. It serves as a central resource for searching across a set of search peers.
The search heads in a cluster are, for most purposes, interchangeable. All search heads have access to the same set of search peers. They can also run or access the same searches, dashboards, knowledge objects, and so on.
A search head cluster is the recommended topology when you need to run multiple search heads across the same set of search peers. The cluster coordinates the activity of the search heads, allocates jobs based on the current loads, and ensures that all the search heads have access to the same set of knowledge objects.
Horizontal scaling for enhanced performance
High availability
No single point of failure
Members share:
Job scheduling.
Search artifacts - search results from scheduled searches
Configurations
captain, which means that it coordinates job scheduling and replication activities among all the members.
If the cluster lacks a majority of members and therefore cannot elect a captain, the members will continue to function as independent search heads. However, they will only be able to service ad hoc searches. Scheduled reports and alerts will not run, because, in a cluster, the scheduling function is relegated to the captain. In addition, configurations and search artifacts will not be replicated during this time.
Capitan election - min 51% votes from all members
The cluster maintains multiple copies of search artifacts resulting from scheduled saved searches. The replication factor determines the number of copies that the cluster maintains of each artifact. For example, if the replication factor is three, the cluster maintains three copies of each artifact: one on the member that originated the artifact, and two on other members. The captain coordinates the replication of artifacts to cluster members. As with any search head, clustered or not, when a search is complete, its search artifact is placed in the dispatch directory of the member originating the search. The captain then directs the artifact's replication process, in which copies stream between members until copies exist on the replication factor number of members, including the originating member.
KVStore
The app key value store (or KV store) provides a way to save and retrieve data within your Splunk apps
In a search head cluster, when a node receives a write request, the KV store delegates the write to the KV store captain