It seems like your browser didn't download the required fonts. Please revise your security settings and try again.
Barracuda CloudGen Firewall

This Firmware Version Is End-Of-Support

Documentation for this product is no longer updated. Please see End-of-Support for CloudGen Firewall Firmware for further information on our EoS policy.

Logging of Clock Skew Events

  • Last updated on

Clock skews are events that describe an inconsistency in the timed recording of sequences. For example, this can occur when the system time has been changed, through which the incremental record of the time stamp is disturbed in the log. This article provides an overview on the clock skew mechanism.

Example

The following figure shows a clock skew event in the past. The leap in system time (indicated as red vertical bar) results in the recording of sequence pairs in the log file, which show the same time stamp.

For this reason, if you start to browse the log from an inconsistent starting point (log query start date B, see the question mark in the figure), it is ambiguous which starting point is meant. For this reason, a popup window will appear that lets you decide to choose the log query start date in order of the chronological occurrence of the clock skew entries in the corresponding log.

log_clock_sk.png

Analyzing Clock Skew Entries in Log Files

This overview is meant to explain the cause of the most frequent clock skew entries produced by dstats/dstatm. Particular regard is paid to those messages generated in dirty situations. dstats and dstatm search for clock skews on every daily start-up of the service. The log file entries they produce will be related to the following processes: 

  • Clock skew detection.
  • Synchronization of actively configured polling list and database.
  • Reasons for service start-up abortion.
  • Synchronization between the local copy of the HA database (which has been mirrored during the last HA sync) and the current database of the HA partner.
  • In addition, dstats searches for files, which are outdated due to a clock skew and should have been deleted according to the configuration file. If the checking routine fails, manual action must be taken. 

Some of the errors described below might produce an additional log file entry like "Comment / MAIN ADMIN action required!!!" The reason for such a message is that the main task is not running. Whenever you encounter it, you might have to restart the task manually. In addition, some of the malfunctions described in the following might additionally produce an entry in the CC control > Stat Collect tab. In case the value of these entries is "INTERNAL ERROR", please contact your sales partner or Barracuda CloudGen Firewall Technical Support.

Log File Entries Related to Clock Skew Detection
Log (Type/Message) Corresponding Content of BerkeleyDBHeader Reason

"Info / MAIN no clock skew detection (initial)"

LastRun 0, LastStart 0

A clock skew cannot be detected because it is assumed that dstatm is either running for the first time or it has never run successfully.

"Error / *** Unresolvable clock skew detected ***"

LastRun <timestamp>, today <timestamp>

HASync is active. The current system time is behind the time of the LastRun header field in the BerkeleyDB. A clock skew detection fails because of inconsistencies in time settings (for more information, see: Best Practice - How to Handle Incorrect Time Settings).

HASync is not active. The current system time is either behind or more than two days ahead the time of the LastRun header field in the BerkeleyDB. A clock skew detection fails because of inconsistencies in time settings (for more information, see: Best Practice - How to Handle Incorrect Time Settings ).

Log File Entries Related to Synchronization of Polling List and Database
Log (Type/Message) Reason
-

The configuration may not be able to be read.

"Error / MAIN cannot sync box state with configuration: <specific error message>"

The configuration of the polling list cannot be synchronized with the database.

"Error / MAIN cannot save state db: <specific error message>"

The configuration of the polling list cannot by synchronized with the database.

Log File Entries Related to Service Start-Up Abortion
Log (Type/Message) Reason

"Fatal / MAIN is disabled!"; "Comment / MAIN ADMIN action required!!!"

Main is in state DISABLED.

"Notice / MAIN trying to recover from <current_state> state"

Main is not in state CLEAN. If the field 'main task' has been set to STOPPED, it will now be reset to IDLE. It will be assumed that the problem has been solved and normal operation can be continued.

Log File Entries Related to Synchronization between HA Databases
Scenarios that will stop task MAIN

These scenarios will produce an additional log file entry "Comment / MAIN ADMIN action required!!!". Checking the state of task MAIN is required. 

Log (Type/Message) Reason

"Error / MAIN sync state dirty!"

HASync is active, but the sync state-file is DIRTY.

"Warning / MAIN ha entry: activity state changed to disabled"

HASync is not active, but the database entry for the HA partner has been not set to DISABLED.

"Info / MAIN ha state db not available, assuming initial"

HASync is active, but the database of the HA partner is not available. It is assumed that the HA partner has never been active and thus has no "state", which it could have negotiated during the HASync. The database of the HA partner will contain the comment entry "no state present, assuming initial".

"Error / MAIN cannot load HA state db: <specific error message>"

HASync is active but the database of the HA-partner is though available not readable.

"Error / MAIN HA state db out of date"

HASync is active and the database of the HA partner is available. There are time inconsistencies in the LastStart header fields of the BerkeleyDBs though, which means the LastStart header field in the own BerkeleyDB is younger than the one in the HA partner's DB. In addition, the DB header field LastRun does not reflect the current day.

The database of the HA partner is obsolete. A data inconsistency is most likely. Since an automated troubleshooting is not possible in this case, a manual check must be undertaken. Possible scenario: The active HA partner has crashed during HA-synchronization. The following log entry could be expected in this case (see next entry below):

"Info / MAIN HA state db out of date? Assuming block and restart scenario"

HASync is active but the ActivityState of the HA partner is DISABLED.

"Warning / MAIN HA sync enabled although HA box is disabled"

HASync is active but the ActivityState of the HA partner is DISABLED.

"Fatal / AIN HA takeover in inconsistent state!"

HaSync is active but the HA partner is not in state CLEAN.

"Warning / MAIN ha entry: activity state changed to disabled"

HaSync is not active but the HA partner state is not DISABLED.

"Error / MAIN session state unknown, going to stop!", "Error / MainLoop - main task UNKNOWN"

The state of MAIN could not be determined during start-up.

"Error / MAIN cannot sync poll state, going to stop!", "Error / MainLoop - main task poll_pending"

Synchronization of configured polling list and database is not possible.

"Error / MAIN internal error <error_num>", "Error / MainLoop - main task poll_pending"

A system related error has occurred during polling (for example missing system resources)

Scenarios that will not stop the task MAIN

The errors described below will not stop task MAIN because there will be no indication that data on the (local) MAIN has been damaged. Take into consideration that, on the other hand, data on the HA partner could be in an inconsistent mode. In case the synchronization is not successful, the current try is given up and the MAIN task is reset to 'sync_await'. When the maximum allowed number of retries is exceeded, the main status changes to 'await_daybreak'. The database will not be synchronized with the HA partner. Manual action will be necessary to solve the problem. 

Log (Type/Message) Reason

"Error / MAIN local compression cooking done with error %d, going to stop!", "Error / MainLoop - main task cook_pending"

Cooking of statistics files could not be completed.

"Error / MAIN cannot write HA sync file", "Error / MainLoop - main task sync_pending"

The HA sync file could not be written. Check for a previously created HA sync file which possibly could not be overwritten. check for HDD errors. Restart dstatm.

"Error / MAIN HA sync done with error <error_num>", "Comment / MAIN could not start sync process"

The sync process cannot be started.

"Error / MAIN HA sync unsuccessful (try <retry_count>)"

The HA synchronization has failed. Prior error messages are to be analyzed to solve this problem.