Security Logs: Cryptographically Signed Audit Logging for Data Protection

In this article, we cover cryptographically signed audit logging, aka “secure logging”, when logs are generated in a certain way which prevents tampering messages, removing, adding, or changing the order of log entries. We explain why signed logs are essential for security software, how we’ve built-in secure audit logging in Acra, and how to use it together with other defense-in-depth layers in your systems.

What Is Cryptographically Signed Audit Logging?

Audit log, aka “audit trail”, is a set of security-relevant log records in a chronological order, which identifies a source and a reason of a log, and provides relevant context for further investigation. 

Audit logs contain information about access management events (like getting or revoking access, access errors), sensitive actions made by users (deleting data), events from services, and critical errors. 

The purpose of the audit log is to provide documentary evidence of the sequence of activities and help to reconstruct what happened (as a response to an incident or regulator inquiry).

As audit logs become the security source of truth, they require protection. In many cases, audit logs are available to superusers only, stored in a special storage, and backed up.

However, as logs are just text messages, they require a set of security procedures (see NIST SP 800-92) and often become an easy target to manipulate and tamper.

When and Why You Might Need Cryptographically Signed Audit Logs

The ability to generate logs in a tamper-free way and then efficiently verify all (or some) log entries are vital to any application employing secure logging techniques. A cryptographically signed audit log is a set of log entries, protected from modification with a help of cryptographic binding. The logging process and log entries structure is designed to mitigate the possibility of making any unnoticed adversarial changes in the logs.

Having a cryptographic validation of each log entry makes it possible to verify the whole log chain and find the first log message that was tampered.

Precise identification of tampered log entries helps to understand when a system was compromised, define trust scope and reconstruct the attack.

How We Implemented Secure Audit Trail In Acra

Now let's move to practical experience. Here we cover how we implemented a secure audit trail in the Acra database protection suite that provides tools for protecting data stored in databases, like “on the fly” encryption and decryption, data pseudonymization and masking, search through encrypted data, etc. Acra logs contain information about application and database connection attempts, API usage, key management actions, accepting/denying incoming queries (for SQL firewall), data processing events (encryption, decryption, pseudonymization, etc). 

Since Acra suite is intended to provide data security across different parts of infrastructure, we use a complex secure approach and introduce cryptographically signed audit logging for each Acra tool.

All Acra tools that have access to users’ data, generate crypto signed logs. Acra calculates integrity checks for each log entry. Each check depends on the current and previous log entry, thus building a log chain (think about blockchain). Acra provides a separate utility, AL Verifier, that takes a crypto signed log and verifies the authenticity of each log entry moving along the chain.

Security Requirements Towards Cryptographically Signed Audit Logs

While studying various secure logging systems (“A New Approach to Secure Logging” paper had a significant influence on us), we gathered a set of security requirements for crypto signed audit logging:

1. Log messages in each log entry should be protected against tampering (symbols editing).

2. The structure of the log chain should be protected against modification: adding, removing separate log entries. 

3. Log chains should be protected against truncation attacks when logs are truncated from the end of the chain.

4. Crypto signed audit logs should be initialized as early as possible at the startup of the logging service, to reduce the number of unprotected log entries in the log chain.

5. Service restart/shutdown should not lead to inconsistent audit logs. If service was shut down due to the emergency, audit logs should be verifiable.

6. Key security: a cryptographic key (used for computing integrity checks) should be stored in dedicated key storage, and reside minimum time in service’ memory (take a look at aes-finder utility that scans memory for AES keys).

7. Acceptable performance: the ability to verify production volumes of protected logs in seconds. Performance directly affects incident detection and reaction time (MTTD, MTTR), if verification of audit logs is an automated monitored procedure.

8. Log rotation friendliness: audit logs should be compatible with log rotation policies typical for distributed systems.

Cryptographic Scheme For Secure Audit Logging

After carefully analyzing two cryptographic schemes described in the above-mentioned paper, gathering security and usability requirements, we have chosen a scheme based on symmetric keys (“private verifiable” scheme).

This scheme has a simple and boring cryptographic design and higher performance than the second scheme (“public verifiable”). It has its own limitations: using the same key for log generation and verification leads to strict security requirements towards verification service.

The private verifiable scheme is built using two cryptographic primitives. 

The first one is a keyed hash function (HMAC) and the second is a usual hash function (hash). HMAC is configured by asymmetric cryptographic key (k) which is the one and only security secret (one key to rule them all!).

Each audit log entry consists of message, date, action, error code, etc. For each log entry we add integrity check which is computed iteratively by the following formal principle:

As a result, audit log chain will consist from:

• LE[1] ║ IC[1]
• LE[2] ║ IC[2]
• ...
• LE[n] ║ IC[n]

where LE[n] — is the body of the n-th entry in the log stream, IC[n] is the integrity check of the n-th log entry and ║ means concatenation.

The value of IC[n] depends on LE[n] and internal state[n]. The internal state is not presented in a log chain but is kept in memory during log generation/verification. IC[n] value is computed only if the value of k and previous state state[n-1] is known, while IC[n] itself does not reveal information about previous log entries, internal state, etc. 

Implementing Secure Audit Trail

Audit logging usage consists of two stages: producing crypto signed logs and, later, verifying their integrity.

• Producing secure logs

AL Logger is an internal component of Acra tools. That’s the last step of processing logs before output. AL Logger takes each log entry, computes its integrity check, and includes it into the structure of log entry (producing LE[n] ║ IC[n]).

AL Logger uses cryptographic key k to calculate integrity checks. This key is generated by AcraKeymaker during Acra setup and configuration. In the case of multi-instance installation, each Acra tool can use its own log key.

As the key k is required only to generate the first integrity check IC[1] (then the key is re-hashed for each next entry), it stays in process memory for a minimal time.

func (f *LogEntryIntegrityCalculator) calculateHmac(input []byte) []byte {

    h := hmac.New(sha256.New, f.cryptoKey)

    h.Write(input)

    h.Write(f.previousLogEntryIntegrityCheck)

    return h.Sum(nil)

}

Calculating internal state[i] as HMAC of k[i], LE[i] and state[i-1]. 

Acra supports various log rotation policies, so it can be configured to produce smaller log files, resetting the log chain upon some event (by signal or after N log entries). In this case, AL Logger finalizes the current log chain and starts a new one. This approach makes Acra easier to use in high load distributed systems: verifying smaller chains takes less time and is more convenient for log rotation.

This is an example of an audit log stream from one of Acra tools:

The audit log’s integrity checks are highlighted by green lines. Note that the first 5 entries of the log stream do not have integrity checks, while further entries do have them (5th log entry is the start of a new log chain, LE[1] ║ IC[1]). This limitation comes from real-world engineering: once started, the Acra tool performs initialization and self-checks procedures first, then accesses the key storage, and only then it’s ready to produce cryptographically signed logs.

• Verifying Secure Logs

AL Verifier is a separate tool that runs independently of other Acra services.

AL Verifier takes audit log file(s), processes each log entry, calculates expected IC[n], and compares it with the actual IC[n] of the log entry. Processing log entries one-by-one, AL Verifier validates the authenticity and integrity of each log entry, log chain, or the entire log stream (a chain of chains).

AL Verifier can indicate only the line of the first invalid log entry because each integrity check depends on the previous log entry.

AL Verifier requires the same cryptographic key k to validate log entries. As we mentioned earlier, using a symmetric scheme requires AL Verifier to run in a trusted environment. Crypto key k stays in memory for a short time, as it’s necessary only to verify a first log entry.

If AL Verifier finds log entries within the log chain that don’t have integrity checks, it displays warnings or errors on such log entries. It’s possible to configure “treat all warnings as errors” for systems with strict security requirements.

When verifying a list of files, AL Verifier works similarly to a popular logrotate tool, making the adoption of continuous audit log verification as easy as log rotation.

This is the example of successful verification of audit log stream:

As we mentioned previously, several log entries at the beginning don’t have integrity checks because they are part of the initialization procedure. As this is expected behavior and these log entries happened before the start of a new crypto signed log chain, AL Verifier reports them as warnings. All following log entries have valid integrity checks, and verification is successful.

Now, let's take this audit log, modify a log message on the 9th line (we deliberately removed one symbol from the integrity string), and verify it.

As expected, verification fails now:

AL Verifier reported the error and the line number of the first invalid integrity check (caused by a mismatch between the expected value and the actual value). 

Logs Meet Real World

Cryptographically signed audit logs and their verification is one of the security controls in the system which should work together with other security controls, building layers of defense in depth. Previously we explained how Acra can help you to build defense in depth around data, here we outline some practical considerations of protecting systems with secure audit logging.

• Protect log contents

Acra logs are signed, not encrypted because they don’t contain sensitive information (any potentially sensitive information, like query params, is removed by Acra’s log handlers). If your system’s logs contain sensitive data, consider using data encryption for the log files at rest.

• Protect against log deletion

It’s important to remember that cryptography helps to verify the authenticity of crypto signed audit logs, but it can’t actually prevent attackers from deleting log entries or log files themselves. Secure audit logs help to find invalid log entries but don’t help to identify what exactly had happened based on the log itself.

It makes sense to use a combination of security controls to ensure that logs are untouched: using special storage devices (like WORM drives), ensuring multiple backups in the physically independent locations, monitoring sudden changes in log files’ size, etc.

• Protect against attacks on logging producers and verifiers

If attackers gain control of logging or verifying service (i.e., one of the Acra tools), they might affect produced logs or results of verification.

Acra tools should run inside a trusted perimeter on a host machine protected by several security measures: access control, monitoring, repeated authentication. 

It makes sense to configure an intrusion detection system on the host machine which checks that the audit log file has been successfully stored, copied, or verified. (H)IDS should alert on errors while creating/copying files, and on log verification failures.

• Automate log verification

Use log verification as an ongoing automated procedure (verify logs every X hours, every Y days, or on log rotate event). Ongoing verification of audit logs helps to detect potential issues earlier. Acra can be configured to reset the log chain and start a new one according to a specific schedule.

• Log backups

Configure audit log backups into different locations. Depending on the industry and regulations, backups can be required to be stored for a long time.

• Log key rotation

The log key should be periodically updated/rotated using a remote trusted server or local trusted hardware. This should be a part of regular key management procedures run in the system. Note that rotated log keys might need to be stored to verify old audit log files.

Open Sourcing Audit Logging Module

Currently cryptographically signed audit logging is available in Acra Enterprise Edition only. We have plans to open-source the audit logging module as a separate library (AL Logger and AL Verifier) later. The audit logging library will fulfil our collection of free and open-source security tools that help developers to protect their users’ data.

Closing Thoughts

Cryptographically signed audit logs are one of the ways of protecting the integrity and authenticity of audit logs. We use a cryptographic scheme where the integrity check of each log entry depends on the previous log entry. Because of the chaining, verification of the audit log file shows if the whole log stream is valid, and indicates the first invalid log entry for tampered logs.

Combining crypto signed audit logging with ongoing automated verification shortens MTTD and MTTR on potential incidents.

I believe that companies and users should own their data. Therefore, we strongly recommend using fast, reliable, and secure tools for data protection that work inside your system.

Kudos to the Cossack Labs' engineering team and Artem Storozhuk, who made this technology and post possible.