Non-repudiative logging

Non-repudiation is a very interesting subject at the intersection of technology and law. For the sake of exercise let’s imagine a very simple scenario in which we’d like to have some specific user actions logged into a file in a way that would allow us to find out later, whether anybody has modified the content of such a log file. Let’s naively assume that anything that the application puts to logs can be trusted.

In order to accomplish this task, we will create a Log4j appender that will calculate an HMAC (hash) of every logged event and attach it to the log file as well.

This would be enough to ensure that any change to a log entry can be spotted. But since we also want to be safe against having log entries removed from the file, we need to create a relation between consecutive entries. In order to do that – each log entry hash will be calculated based on its content as well as the previous entry’s hash value. Removal of any entry will then ‘break the chain’ and result with wrong hashes of all the following ones.

For the solution to be complete, it is necessary to initialize the hash generator with a secret salt value – otherwise a person modifying our data could easily recalculate all hashes from the beginning and bypass the whole mechanism. The salt should be known only to the appender calculating hashes and to the third party willing to validate integrity of the log file. This of course assumes that the third party is trustworthy – in some more sophisticated scenarios you might want to use a public key infrastructure to avoid using the same secret by the two parties.

The code below consists of five files:

DigestAppender.java – the appender itself, based on parameters provided by Log4j configuration will write all log events with their hashes to System.out (for the sake of simplicity)

Test.java – a simple class logging a few events.

Validator.java – a tool that will verify the integrity of user-provided log file and also output it’s content stripped of lines containing hashes (so it’s easier to read)

Util.java – container for some shared pieces of code.

log4j.properties – a simple configuration specifying (among others) a hashing algorithm (SHA-256 in this case) and the salt etc.

Example usage:

java -cp .;./lib/log4j-1.2.15.jar Test > output.log
java -cp .;./lib/log4j-1.2.15.jar Validator output.log SHA-256 a$ecret$eed

DigestAppender.java:

import java.security.MessageDigest;

import org.apache.log4j.AppenderSkeleton;
import org.apache.log4j.spi.LoggingEvent;

public class DigestAppender extends AppenderSkeleton {

	private String algorithm;
	private transient String seed;
	private transient MessageDigest messageDigest;

	public void append(LoggingEvent event) {
		String eventString = layout.format(event) + Util.LINE_SEPARATOR;
		String digest = Util.digestAndUpdate(eventString, getMessageDigest());
		System.out.println(eventString.length() + "," + digest);
		System.out.print(eventString);
	}

	private MessageDigest getMessageDigest() {
		if (messageDigest == null) {
			try {
				messageDigest = MessageDigest.getInstance(algorithm);
				Util.digestAndUpdate(seed, messageDigest);
				seed = null; // will not be needed anymore
			}
			catch (Exception e) {
				e.printStackTrace();
			}
		}
		return messageDigest;
	}

	public boolean requiresLayout() {
		return true;
	}

	public void setAlgorithm(String algorithm) {
		this.algorithm = algorithm;
	}

	public void setSeed(String seed) {
		this.seed = seed;
	}

	public void close() {
	}
}

Test.java:

import org.apache.log4j.Logger;

public class Test {

	private static Logger log = Logger.getLogger(Test.class);

	public static void main(String[] args) {
		new Test().run(args);
	}

	public void run(String[] args) {

		for (int i = 0; i < 10; i++) {
			log.info( "This is an info message number " + i);
			log.debug("This is a debug message number " + i);
			log.warn( "This is a warn  message number " + i);
		}
	}
}

Validator.java:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.security.MessageDigest;

public class Validator {

	protected MessageDigest messageDigest;

	public static void main(String[] args) throws Exception {
		if (args.length < 3) {
			throw new Exception("Usage: [logFile] [algorithm] [seed]");
		}
		new Validator().validate(new File(args[0]), args[1], args[2]);
	}

	public void validate(File logFile, String algorithm, String seed) throws Exception {
		messageDigest = MessageDigest.getInstance(algorithm);
		Util.digestAndUpdate(seed, messageDigest);
		processFile(logFile);
	}

	public void processFile(File file) throws Exception {
		BufferedReader br = null;
		try {
			br = new BufferedReader(new FileReader(file));
			while (true) {
				String line = br.readLine();
				if (line == null) {
					break;
				}
				String[] entryDetails = line.split(",");
				int logEntryLength = Integer.parseInt(entryDetails[0]);
				String expectedDigest = entryDetails[1];
				char[] logEntryBuffer = new char[logEntryLength];
				if (br.read(logEntryBuffer) == -1) {
					break;
				}
				validateLogEntry(String.valueOf(logEntryBuffer), expectedDigest);
			}
		}
		finally {
			if (br != null) br.close();
		}
	}

	protected void validateLogEntry(String logEntry, String expectedDigest) throws Exception {
		String digest = Util.digestAndUpdate(logEntry, messageDigest);
		if (digest.equals(expectedDigest) == false) {
			throw new Exception("Wrong checksum of entry:" + Util.LINE_SEPARATOR + logEntry);
		}
		System.out.print(logEntry);
	}
}

Util.java:

import java.security.MessageDigest;

public class Util {

	public static final String LINE_SEPARATOR = System.getProperty("line.separator");

	public static String digestAndUpdate(String input, MessageDigest messageDigest) {
		try {
			byte[] digestResult = messageDigest.digest(input.getBytes());
			messageDigest.update(digestResult);

			StringBuilder hexResult = new StringBuilder();
			for (byte b : digestResult) {
				String hexByte = Integer.toHexString(0xFF & b);
				if (hexByte.length() == 1) {
					hexResult.append("0");
				}
				hexResult.append(hexByte);
			}
			return hexResult.toString();
		}
		catch (Exception e) {
			e.printStackTrace();
			return null;
		}
	}
}

log4j.properties:

log4j.rootLogger=INFO, digestAppender
log4j.appender.digestAppender=DigestAppender
log4j.appender.digestAppender.algorithm=SHA-256
log4j.appender.digestAppender.seed=a$ecret$eed
log4j.appender.digestAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.digestAppender.layout.ConversionPattern=[%d{MMM dd HH:mm:ss}] %-5p (%F:%L) - %m
log4j.appender.digestAppender.threshold=INFO
log4j.category.Test=INFO
log4j.logger.Test=digestAppender

Finally, here is the example log file:

75,68d724760f6892dec0b1612848e4ae1665af05254ea0715a5dc09423d74306ae
[sep 07 15:04:28] INFO  (Test.java:14) - This is an info message number 0
75,0703a6444f0704a66fb14ed87a718ebf821f23fccd375dd5ffbe56da704fc902
[sep 07 15:04:28] WARN  (Test.java:16) - This is a warn  message number 0
75,59c5d9319632139389e71b522d74ca17112b67fda8ba3a5dd7ace692d6c024b6
[sep 07 15:04:28] INFO  (Test.java:14) - This is an info message number 1
75,937edac9d2570c0c6d1c5083145971305b21f0ebd00fbb23abf9862646867284
[sep 07 15:04:28] WARN  (Test.java:16) - This is a warn  message number 1
75,a05e8219cda5e2f95b9846f5a98d7654713950ab71a4960aec32364024391ed4
[sep 07 15:04:28] INFO  (Test.java:14) - This is an info message number 2
75,479010bf0ebd11031733e035a1aa793b38887fe91201e534b3cfd4678778af31
[sep 07 15:04:28] WARN  (Test.java:16) - This is a warn  message number 2

Enjoy!