Configuring xMatters web server logging

An xMatters web server comprises three web applications (the web user interface, web services, and the mobile access component), which run within a single Jetty-based application server. The application server and each of the web applications use log4j to produce and record log messages.

Note:

In this article, <xMHOME> refers to the directory where xMatters is installed. For example, the default installation folder on Windows is C:\Program Files\xMatters.

Summary

Each web server uses the following logging directories and logging configuration files:

Critical and startup logging (stderr/stdout)

  • Location: <xMHOME>/logs
  • Configuration files: none

Webserver request logging (HTTP GET/POST)

  • Location: <xMHOME>/webserver/logs
  • Configuration files:
    • <xMHOME>/webserver/etc/jetty.xml
    • <xMHOME>/webserver/contexts/alarmpoint.xml
    • <xMHOME>/webserver/contexts/api.xml
    • <xMHOME>/webserver/contexts/gateway.xml
    • <xMHOME>/webserver/contexts/static.xml

Application server startup/infrastructure

  • Location:<xMHOME>/webserver/logs
  • Configuration file: <xMHOME>/webserver/resources/log4j.xml

Web User Interface logging

  • Location: <xMHOME>/webserver/webapps/cocoon/WEB-INF/logs
  • Configuration file: <xMHOME>/webserver/webapps/cocoon/WEB-INF/classes/log4j.xml

Web Services logging:

  • Location: <xMHOME>/webserver/webapps/axis2/WEB-INF/logs
  • Configuration file: <xMHOME>/webserver/webapps/axis2/WEB-INF/classes/log4j.xml

Mobile Access logging:

  • Location: <xMHOME>/webserver/webapps/mobilegateway/WEB-INF/logs
  • Configuration file: <xMHOME>/webserver/webapps/mobilegateway/WEB-INF/classes/log4j.xml

Most web server log files, with the exception of the web server request logs, are cleared each time the web server starts. This behaviour can be changed by altering the appenders in each of the log4j configuration files. In particular, change the entry

<param name="Append" value="false" />  

to

<param name="Append" value="true" />  

stderr and stdout System Streams

Because the log4j system is not initialized until partway through the startup sequence, and some third-party libraries used by an web server do not support log4j, error and diagnostic output produced via the stderr and stdout system streams may bypass log4j.

To capture this output, the web server redirects stderr and stdout to the following log files, located in the /logssubfolder of the xMatters installation directory:

  • stderr: AlarmPoint.webserver.out.log
  • stdout: AlarmPoint.webserver.err.log

Note: Two additional log files, webserver.out.log and webserver.err.log, may also be present; these files should be ignored.

These log files should be consulted whenever an xMatters web server fails to start, or while diagnosing a runtime problem that does not appear in any of the log4j log files.

For example, if an attempt is made to start the same instance of a web server twice, the second start will fail because the required server ports are already in use. The AlarmPoint.webserver.err.log records the following error message:

java.net.BindException: Address already in use: JVM_Bind WARN: Not listening on monitor port: 8887  

Web Server Request Logging

A web server processes HTTP GET/POST requests (e.g., for Web UI HTML and images), and web services/SOAP requests (e.g., for AddEvent web service calls). Each of these requests appear in one or more log files, which are located in the /webserver/logs subfolder in the xMatters installation directory. The setting is controlled by the

element in the configuration file specified in the following list:

.request.log:

  • Logs requests to URLs that do not begin with /alarmpoint/ or end with .do or .kont.
  • Configuration file: /etc/jetty.xml

.alarmpoint.request.log:

  • Logs requests to URLs that begin with /alarmpoint/, which are related to xMatters web user interface requests.
  • Configuration file: /contexts/alarmpoint.xml

.api.request.log:

  • Logs requests to URLs that begin with /api/, which are related to xMatters web services requests.
  • Configuration file: /contexts/api.xml

.gateway.request.log

  • Logs requests to URLs that begin with /mg/, which are related to xMatters mobile access requests.
  • Configuration file: /contexts/gateway.xml

.static.request.log

  • Logs requests to URLs that begin with /static/, which are related to web user interface requests.
  • Configuration file: /contexts/static.xml

Note: The configuration file paths are relative to the /webserver subfolder of the xMatters installation directory.

Each log file contains the relevant requests that occurred on a particular date. This date is represented by the tag in the file names and is substituted with the current date in GMT (e.g., 2008_11_19.request.log). Log files older than 90 days are automatically deleted. The settings for each request log can be changed via the listed configuration file.

The .request.log files record all requests to a web server except for the indicated exclusions, while the other log files record specific subsets of web server requests. As a result, the same log file entry may appear in both .request.log and several of the subset log files.

For example, when an attempt is made to log in to xMatters via the URL http://localhost:8888/xmatters, the log files record the following entries:

2008_11_19.request.log:

127.0.0.1 - - [Wed, 19 Nov 2008 19:09:16 GMT] "GET /static/resources/forms/js/forms-lib.js HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:16 GMT] "GET /static/resources/forms/css/forms.css HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/includes/javascript/AlarmPoint.js HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/css/AlarmPoint.css HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/includes/javascript/Calendar/lang/calendar-en.js HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/includes/javascript/Validation/lang/validation-en.js HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/images/logos/alarmpoint/logo_login.gif HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/images/layout/login_background.png HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/images/layout/background-red.png HTTP/1.1" 304 0  

2008_11_19.alarmpoint.request.log:

127.0.0.1 - - [Wed, 19 Nov 2008 19:09:11 GMT] "GET /alarmpoint HTTP/1.1" 302 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:14 GMT] "GET /alarmpoint/signOn.do HTTP/1.1" 200 3362 

2008_11_19.static.request.log:

127.0.0.1 - - [Wed, 19 Nov 2008 19:09:16 GMT] "GET /static/resources/forms/js/forms-lib.js HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:16 GMT] "GET /static/resources/forms/css/forms.css HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/includes/javascript/AlarmPoint.js HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/css/AlarmPoint.css HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/includes/javascript/Calendar/lang/calendar-en.js HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/includes/javascript/Validation/lang/validation-en.js HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/images/logos/alarmpoint/logo_login.gif HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/images/layout/login_background.png HTTP/1.1" 304 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:09:17 GMT] "GET /static/images/layout/background-red.png HTTP/1.1" 304 0  

When an attempt is made to access the xMatters web service WSDL via the URL http://localhost:8888/api/AlarmPointWebService, the log files record the following entries:

2008_11_19.request.log:

127.0.0.1 - - [Wed, 19 Nov 2008 19:35:41 GMT] "GET /api/services/AlarmPointWebService?wsdl HTTP/1.1" 200 297510  

2008_11_19.api.request.log:

127.0.0.1 - - [Wed, 19 Nov 2008 19:35:41 GMT] "GET /api/services/AlarmPointWebService?wsdl HTTP/1.1" 200 297510  

These web server request log files are best used to determine whether a specific resource is available (e.g., is the xMatters mobile access web application running). They can also be used to diagnose web server access issues. For example, if a web server is hosted behind a firewall and requests to access the login page are failing, these log files will show whether the request is actually getting through the firewall to xMatters.

While these log files can be used to diagnose some resource access and security problems, they cannot be used to analyze application level logic. For example, a failure to login because of an incorrect password will not be indicated in these logs as shown by the following log entries:

2008_11_19.alarmpoint.request.log

127.0.0.1 - - [Wed, 19 Nov 2008 19:45:04 GMT] "GET /alarmpoint HTTP/1.1" 302 0  
127.0.0.1 - - [Wed, 19 Nov 2008 19:45:05 GMT] "GET /alarmpoint/signOn.do HTTP/1.1" 200 3362
127.0.0.1 - - [Wed, 19 Nov 2008 19:45:10 GMT] "POST /alarmpoint/signOn.do HTTP/1.1" 200 3459  

The POST entry corresponds to the user clicking the "Sign In" button on the login page. Even though the user entered an incorrect password, at a resource level, the POST to signOn.do is successful since the resource exists (i.e., HTTP 200). At the application level, however the login is unsuccessful, but you would have to consult the Application Logging or the Security Audit Report to determine this.

Application Logging

The Jetty-based application server and each of the web applications have their own log4j configuration. The log4j configuration for the application server as a whole is located at <xMHOME>/webserver/resources/log4j.xml. The default log4j configuration defines the following log files, which are located in <xMHOME>/webserver/logs:

  • main.log: logs WARN and INFO messages
  • errors.log: logs FATAL and ERROR messages
  • debug.log: logs DEBUG and higher messages

These application server log files contain entries that are not specific to one of the three web applications, and are mostly used to diagnose problems with starting an xMatters web server (e.g., Jetty misconfiguration). Since the majority of application level logging occurs within one of the web applications, these log files do not generally contain information about the runtime behaviour of a web server.

The majority of an xMatters web server’s application level logging occurs within one of the following three web applications:

  • Web User Interface: <xMHOME>/webserver/webapps/cocoon
  • Web Services: <xMHOME>/webserver/webapps/axis2
  • Mobile access: <xMHOME>/webserver/webapps/mobilegateway

Within each web application’s directory is a log4j configuration file located in WEB-INF/classes/log4j.xmland a log file directory located in WEB-INF/logs.

Each Web Application creates several log files that contain log entries specific to certain components. However, each web application has the following primary log files:

  • AlarmPoint_WebApp.log: logs DEBUG and higher messages produced by xMatters software (i.e., com.invoqsystems.* and com.alarmpoint.* packages)
  • core.log: logs DEBUG and higher messages produced by xMatters software,WARN and higher messages produced by third party software, and INFO and higher messages produced by the JSP interpreter
  • errors.log: logs ERROR and higher messages
  • hibernate.log: logs WARN and higher messages related to database-to-Java object mapping
  • spring.log: logs WARN and higher messages related to Java object initialization
  • pooling.log: logs WARN and higher messages related to database connection pooling

Similar to web server request logging, some log entries may be duplicated across multiple log files. For example, any WARN log entries that are written to hibernate.log will also be written to core.log.

xMatters Reference

DTN-1372, DTN-1339, SUP-4194, SUP-7275, PRE-2463, JDN-1223

Originally created by Don Clark

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk