Access data

Access logs are a source of data for the web server data type. The logs capture incoming requests for content served by httpd implementations which are commonly called web servers. Examples include Apache, IIS, and NGINX. These can be standalone httpd programs, or can be embedded into other products that are internet facing. Current examples of embedded web services can be found as the administrative pages for devices like switches and routers and in portals for cloud based services like Bitbucket, GitHub, and AWS. The format of access logs is governed by several organizations such as  the NCSA and the W3C. In the Common Information Model, access logs can be mapped to any of the following data models: Web.

Data visibility 

Data in these logs provide information about the network address of clients that request content, date and time stamps, request type, status codes, bytes returned, and sometimes username. There is also session information, which is a rich source of information about interactions with the web server and a wide range of applications.

Data application

When your Splunk deployment is ingesting Access logs, you can use the data to achieve the following objectives:

• Managing web server performance
• Monitoring use of Git repositories
• Reconstructing a website defacement

Configuration

The following sections provide information on configuring Splunk software to ingest this data source. To configure the device or software, we recommend that you leverage official web server (e.g. Apache, IIS, or NGINX) resources.

Getting Access Log data in

If your deployment is not already ingesting Carbon Black data, follow the Getting Data In guidance for Splunk Enterprise or the Onboarding and Forwarding Your Data guidance for Splunk Cloud.

The recommended index is web. 

The source type is *:access* or ms:*:*. 

The supported input types are monitors.

In addition, you will need one of the several add-ons for your web server. The most common are listed below. Read and follow the documentation carefully to understand all the essential information you need to work with this data source, including how to install the add-on, configure Web Server logs, and configure Splunk. 

Sizing estimate

The best way to estimate sizing is to send the data to Splunk and use the monitoring console to get ingest sizing by index or sourcetype. Data ingest will vary widely, but an estimated baseline is 500/MB per day per server.

Validation

Validation is done by searching the index and validating timestamp, sourcetype, and field extractions. A search similar to the following is a good starting point. You can limit the search to the index you configured by adding your choice of name e.g., index=foo.

| tstats values(sourcetype) AS type WHERE index=_* group BY index 
| search type=*access OR type=ms*