This is definitely not desired – in case of specific input that contains repeatable sets of information (like text documents, bitmaps with large areas of the same color), the encrypted chunks might repeat just like original plain-text values do. As a result of that, encrypted data might reveal important details of the original content.

You might want to have a look at a very interesting article about this subject on (ahem) Wikipedia. It clearly describes, how block cipher modes solve the problem and additionaly demonstrates how it manifests itself in case of bitmaps. I wanted to play with various block cipher modes and in order to do so I wrote a simple tool that encrypts a user-provided BMP file while leaving its headers intact. As a result of that we are still able to view the encrypted bitmap file in image browser and observe how particular ciphers, key sizes and block modes influence the output. Below are some examples showing:

1. raw image
2. image encrypted with 56 bit DES using Electronic Codebook (ECB) mode
3. image encrypted with 56 bit DES using Cipher-block Chaining (CBC) mode

In case of the second example I chose a rather high resolution (3923×4656) scan of an old document. What you see here is just a small cropped part of it, which is hopefully more than enough to observe the effect. I guess I could spare us all pictures representing the CBC mode since they are pretty noisy.

We can also look at this subject from slightly different perspective. The chart below shows how byte values are distributed in a file. Horizontal axis represents bytes from 0 to 255 obviously, while vertical axis stands for count of bytes of given value.

You can clearly see that in case of raw data as well as ECB mode some bytes show up more often than others. But if we look at the green line representing CBC mode – the distribution is very even across the whole scale which suggests that the encrypted data would contain less information valuable to a person willing to find out what is behind it.

The code to the BMP encrypter is below.

import java.io.*;
import java.security.GeneralSecurityException;

import javax.crypto.*;
import javax.crypto.spec.IvParameterSpec;

public class BitmapPaddingTest {

    public static void main(String[] args) throws Exception {
        if (args.length < 4) {
            throw new Exception("Parameters: [file] [cipher] [keySize] [useInitVector]");
        }
        new BitmapPaddingTest().encrypt(
                new File(args[0]), args[1], Integer.parseInt(args[2]), Boolean.parseBoolean(args[3]));
    }

    public void encrypt(File file, String cipherName, int keySize, boolean useIv) throws Exception {
        File encryptedFile = buildEncryptedFile(file, cipherName, keySize);
        System.out.print("Encrypting " + file.getName() + " to " + encryptedFile.getName() + "... ");

        OutputStream outputStream = new BufferedOutputStream(new FileOutputStream(encryptedFile));
        InputStream fileStream = new BufferedInputStream(new FileInputStream(file));
        byte[] data = new byte[54]; // first 54 bytes of BMP file contain its header (in most cases)
        fileStream.read(data);
        outputStream.write(data);
        outputStream.flush();        

        CipherOutputStream cipherStream =
            new CipherOutputStream(outputStream, createCipher(cipherName, keySize, useIv));
        data = new byte[1024];
        int size = 0;
        while ((size = fileStream.read(data)) > 0) {
            cipherStream.write(data, 0, size);
        }
        cipherStream.flush();
        cipherStream.close();
        System.out.println("done.");
    }

    private SecretKey createSymmetricKey(String cipherName, int keySize) throws GeneralSecurityException {
        KeyGenerator keyGenerator = KeyGenerator.getInstance(cipherName.substring(0, cipherName.indexOf("/")));
        keyGenerator.init(keySize);
        return keyGenerator.generateKey();
    }

    private Cipher createCipher(String cipherName, int keySize, boolean useIv) throws GeneralSecurityException {
        SecretKey symmetricKey = createSymmetricKey(cipherName, keySize);
        Cipher cipher = Cipher.getInstance(cipherName);
        IvParameterSpec iv = useIv ? new IvParameterSpec(symmetricKey.getEncoded()) : null;
        cipher.init(Cipher.ENCRYPT_MODE, symmetricKey, iv);
        return cipher;
    }

    private File buildEncryptedFile(File file, String cipherName, int keySize) {
        String cipherNameUnderscored = cipherName.replaceAll("/", "_");
        return new File(file.getParentFile(),
                file.getName().replace(".bmp", "_" + cipherNameUnderscored + "_" + keySize + ".bmp"));
    }
}

If run as follows:

java BitmapPaddingTest source.bmp DES/ECB/PKCS5Padding 56 false
java BitmapPaddingTest source.bmp DES/CBC/PKCS5Padding 56 true

will generate two files located in the same directory as the source bitmap.

In order to accomplish this task, we will create a Log4j appender that will calculate an HMAC (hash) of every logged event and attach it to the log file as well.

This would be enough to ensure that any change to a log entry can be spotted. But since we also want to be safe against having log entries removed from the file, we need to create a relation between consecutive entries. In order to do that – each log entry hash will be calculated based on its content as well as the previous entry’s hash value. Removal of any entry will then ‘break the chain’ and result with wrong hashes of all the following ones.

For the solution to be complete, it is necessary to initialize the hash generator with a secret salt value – otherwise a person modifying our data could easily recalculate all hashes from the beginning and bypass the whole mechanism. The salt should be known only to the appender calculating hashes and to the third party willing to validate integrity of the log file. This of course assumes that the third party is trustworthy – in some more sophisticated scenarios you might want to use a public key infrastructure to avoid using the same secret by the two parties.

The code below consists of five files:

DigestAppender.java – the appender itself, based on parameters provided by Log4j configuration will write all log events with their hashes to System.out (for the sake of simplicity)

Test.java – a simple class logging a few events.

Validator.java – a tool that will verify the integrity of user-provided log file and also output it’s content stripped of lines containing hashes (so it’s easier to read)

Util.java – container for some shared pieces of code.

log4j.properties – a simple configuration specifying (among others) a hashing algorithm (SHA-256 in this case) and the salt etc.

Example usage:

java -cp .;./lib/log4j-1.2.15.jar Test > output.log
java -cp .;./lib/log4j-1.2.15.jar Validator output.log SHA-256 a$ecret$eed

DigestAppender.java:

import java.security.MessageDigest;

import org.apache.log4j.AppenderSkeleton;
import org.apache.log4j.spi.LoggingEvent;

public class DigestAppender extends AppenderSkeleton {

	private String algorithm;
	private transient String seed;
	private transient MessageDigest messageDigest;

	public void append(LoggingEvent event) {
		String eventString = layout.format(event) + Util.LINE_SEPARATOR;
		String digest = Util.digestAndUpdate(eventString, getMessageDigest());
		System.out.println(eventString.length() + "," + digest);
		System.out.print(eventString);
	}

	private MessageDigest getMessageDigest() {
		if (messageDigest == null) {
			try {
				messageDigest = MessageDigest.getInstance(algorithm);
				Util.digestAndUpdate(seed, messageDigest);
				seed = null; // will not be needed anymore
			}
			catch (Exception e) {
				e.printStackTrace();
			}
		}
		return messageDigest;
	}

	public boolean requiresLayout() {
		return true;
	}

	public void setAlgorithm(String algorithm) {
		this.algorithm = algorithm;
	}

	public void setSeed(String seed) {
		this.seed = seed;
	}

	public void close() {
	}
}

Test.java:

import org.apache.log4j.Logger;

public class Test {

	private static Logger log = Logger.getLogger(Test.class);

	public static void main(String[] args) {
		new Test().run(args);
	}

	public void run(String[] args) {

		for (int i = 0; i < 10; i++) {
			log.info( "This is an info message number " + i);
			log.debug("This is a debug message number " + i);
			log.warn( "This is a warn  message number " + i);
		}
	}
}

Validator.java:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.security.MessageDigest;

public class Validator {

	protected MessageDigest messageDigest;

	public static void main(String[] args) throws Exception {
		if (args.length < 3) {
			throw new Exception("Usage: [logFile] [algorithm] [seed]");
		}
		new Validator().validate(new File(args[0]), args[1], args[2]);
	}

	public void validate(File logFile, String algorithm, String seed) throws Exception {
		messageDigest = MessageDigest.getInstance(algorithm);
		Util.digestAndUpdate(seed, messageDigest);
		processFile(logFile);
	}

	public void processFile(File file) throws Exception {
		BufferedReader br = null;
		try {
			br = new BufferedReader(new FileReader(file));
			while (true) {
				String line = br.readLine();
				if (line == null) {
					break;
				}
				String[] entryDetails = line.split(",");
				int logEntryLength = Integer.parseInt(entryDetails[0]);
				String expectedDigest = entryDetails[1];
				char[] logEntryBuffer = new char[logEntryLength];
				if (br.read(logEntryBuffer) == -1) {
					break;
				}
				validateLogEntry(String.valueOf(logEntryBuffer), expectedDigest);
			}
		}
		finally {
			if (br != null) br.close();
		}
	}

	protected void validateLogEntry(String logEntry, String expectedDigest) throws Exception {
		String digest = Util.digestAndUpdate(logEntry, messageDigest);
		if (digest.equals(expectedDigest) == false) {
			throw new Exception("Wrong checksum of entry:" + Util.LINE_SEPARATOR + logEntry);
		}
		System.out.print(logEntry);
	}
}

Util.java:

import java.security.MessageDigest;

public class Util {

	public static final String LINE_SEPARATOR = System.getProperty("line.separator");

	public static String digestAndUpdate(String input, MessageDigest messageDigest) {
		try {
			byte[] digestResult = messageDigest.digest(input.getBytes());
			messageDigest.update(digestResult);

			StringBuilder hexResult = new StringBuilder();
			for (byte b : digestResult) {
				String hexByte = Integer.toHexString(0xFF & b);
				if (hexByte.length() == 1) {
					hexResult.append("0");
				}
				hexResult.append(hexByte);
			}
			return hexResult.toString();
		}
		catch (Exception e) {
			e.printStackTrace();
			return null;
		}
	}
}

log4j.properties:

log4j.rootLogger=INFO, digestAppender
log4j.appender.digestAppender=DigestAppender
log4j.appender.digestAppender.algorithm=SHA-256
log4j.appender.digestAppender.seed=a$ecret$eed
log4j.appender.digestAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.digestAppender.layout.ConversionPattern=[%d{MMM dd HH:mm:ss}] %-5p (%F:%L) - %m
log4j.appender.digestAppender.threshold=INFO
log4j.category.Test=INFO
log4j.logger.Test=digestAppender

Finally, here is the example log file:

75,68d724760f6892dec0b1612848e4ae1665af05254ea0715a5dc09423d74306ae
[sep 07 15:04:28] INFO  (Test.java:14) - This is an info message number 0
75,0703a6444f0704a66fb14ed87a718ebf821f23fccd375dd5ffbe56da704fc902
[sep 07 15:04:28] WARN  (Test.java:16) - This is a warn  message number 0
75,59c5d9319632139389e71b522d74ca17112b67fda8ba3a5dd7ace692d6c024b6
[sep 07 15:04:28] INFO  (Test.java:14) - This is an info message number 1
75,937edac9d2570c0c6d1c5083145971305b21f0ebd00fbb23abf9862646867284
[sep 07 15:04:28] WARN  (Test.java:16) - This is a warn  message number 1
75,a05e8219cda5e2f95b9846f5a98d7654713950ab71a4960aec32364024391ed4
[sep 07 15:04:28] INFO  (Test.java:14) - This is an info message number 2
75,479010bf0ebd11031733e035a1aa793b38887fe91201e534b3cfd4678778af31
[sep 07 15:04:28] WARN  (Test.java:16) - This is a warn  message number 2

Enjoy!

The interesting thing is that with a little bit of efford it’s actually quite easy to extract the decrypted values from config files by using a few undocumented APIs buried in WebLogic itself. Below you will find a simple tool that will output a content of either .xml or .properties configuration file provided as input, replacing all encrypted values with their original content. It works with WebLogic 10 but can be easily adapted for versions 8 and 9 as well (domain directory structure is slightly different between these versions). I wrote it some time ago in order to be able to easily retrieve logins and passwords of development domains that I kept forgetting too often. Obviously there are many other uses, like configuration management, application server migration tools etc.

You might get worried that since it’s so easy to get to this supposedly safe data then your production environments are endangered. It isn’t so – the decryption mechanism is useless without a file SerializedSystemIni.dat which contains the 3DES key necessary to decrypt the data and is uniqe for every domain. If this file is properly protected from unauthorized access on OS level then have no worries.

(I was going to put here a link to a interesting blog entry on dev2dev.bea.com portal with some insightful comments on this subject but it seems that Dev2Dev has been shut down due to the BEA acquisition by Oracle. I will update this entry if it pops up somewhere else.)

Update: I have noticed that this article is getting quite many views every day – it seems that forgetting domain passwords is more common problem than it seems . Some users reported issues while processing their configuration files, which often is caused by the poor use of REGEXP in my original code. In order to make it easy to all of us I have prepared a new version of the tool. It uses XPATH to find all XML nodes and attributes containing encrypted data. I hope that this approach is more robust.

Update 2: If you are concerned about the risk of accessing the configuration data by deployed applications, then please have a look at comments #32 – #34.

import java.util.*;
import java.io.*;
import javax.xml.parsers.*;
import javax.xml.xpath.*;
import org.w3c.dom.*;

import weblogic.security.internal.*; // requires weblogic.jar in the class path
import weblogic.security.internal.encryption.*;

public class WebLogicDecryptor {

	private static final String PREFIX = "{3DES}";
	private static final String XPATH_EXPRESSION
		= "//node()[starts-with(text(), '" + PREFIX + "')] | //@*[starts-with(., '" + PREFIX + "')]";

	private static ClearOrEncryptedService ces;

	public static void main(String[] args) throws Exception {
		if (args.length < 2) {
			throw new Exception("Usage: [domainDir] [configFile]");
		}

		ces = new ClearOrEncryptedService(SerializedSystemIni.getEncryptionService(new File(args[0]).getAbsolutePath()));
		File file = new File(args[1]);
		if (file.getName().endsWith(".xml")) {
			processXml(file);
		}
		else if (file.getName().endsWith(".properties")){
			processProperties(file);
		}
	}

	private static void processXml(File file) throws Exception {
		Document doc = DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(file);
		XPathExpression expr = XPathFactory.newInstance().newXPath().compile(XPATH_EXPRESSION);
		NodeList nodes = (NodeList)expr.evaluate(doc, XPathConstants.NODESET);
		for (int i = 0; i < nodes.getLength(); i++) {
			Node node = nodes.item(i);
			print(node.getNodeName(), node.getTextContent());
		}
	}

	private static void processProperties(File file) throws Exception {
		Properties properties = new Properties();
		properties.load(new FileInputStream(file));
		for (Map.Entry p : properties.entrySet()) {
			if (p.getValue().toString().startsWith(PREFIX)) {
				print(p.getKey(), p.getValue());
			}
		}
	}

	private static void print(Object attributeName, Object encrypted) {
		System.out.println("Node name: " + attributeName);
		System.out.println("Encrypted: " + encrypted);
		System.out.println("Decrypted: " + ces.decrypt((String)encrypted) + "\n");
	}
}

Feel free to comment and improve.

Fortunately log4j provides a very convenient way of solving this problem. A class org.apache.log4j.MDC provides a mechanism to ‘attach’ any arbitrary information to a given thread that processes a request and to output this information into the log entry. What we need to do is to invoke a static method MDC.put(“userId”, userId) somewhere at the beginning of request processing chain (by using servlet filter for example) and add the element %X{userId} to the row format definition in log4j configuration, for example:

log4j.appender.file.layout.ConversionPattern=%d %5p %c %X{userId} (%F:%L) – %m%n

After that every log entry will contain user identifier which is all we need in most cases. Sometimes we might want to put into the log some additional data. If our application uses AJAX a lot, then a single user might issue multiple requests at the same time, which still can get mixed. It might be a good idea then to add a unique index value incremented after each request etc.

If you add this feature to your application, the people who routinely have to analyze its logs and solve user problems will be most grateful!

long t = System.currentTimeMillis();
doSomething();
System.out.println("took " + (System.currentTimeMillis() - t) + "ms");

There are definitely many cons to this approach – it’s not elegant, the results are scattered all over the log file, it can not be turned off and so on. If you happen to be using Spring Framework or Apache Commons, you might use one of their implementations of StopWatch class (here and here), which is a wrapper with some additional capabilities, like report formatting, providing of statistics etc.

Some time ago I was in a need to use a stop watch for a web application based on Spring. After spending some time with mentioned implementations it became obvious that what I needed was a slightly different mechanism – one that would allow me to see a hierarchical structure of time consumption of individual pieces of code. Something like:

run() : 2753 ms
	foo() : 2753 ms
		bar() : 417 ms
			bang() : 60 ms
			bang() : 119 ms
			bang() : 238 ms
		bar() : 2336 ms
			bang() : 477 ms
			bang() : 953 ms
			bang() : 906 ms

At the end of this post you will find a source code of the class that I came up with. I hope that it might be useful either as it is, or as basis to a more sophisticated implementation. The output looks very much like the example above. The interesting thing about this particular implementation is that since it uses ThreadLocal under the hood, it can be accessed through its static methods. Example usage might look like that:

public class StopWatchTest {

	public static void main(String[] args) {
		StopWatch.reset();
		new StopWatchTest().run();
		System.out.println(StopWatch.prettyPrint());
	}

	public void run() {
		StopWatch.start("run()");
		foo();
		StopWatch.stop();
	}

	public void foo() {
		StopWatch.start("foo()");
		bar();
		bar();
		StopWatch.stop();
	}

	public void bar() {
		StopWatch.start("bar()");
		bang();
		bang();
		bang();
		StopWatch.stop();
	}

	public void bang() {
		StopWatch.start("bang()");
		sleepRandomly();
		StopWatch.stop();
	}

	public void sleepRandomly() {
		try {
			Thread.sleep(System.currentTimeMillis() % 1000);
		}
		catch (Exception e) {
			e.printStackTrace();
		}
	}
}

You can also easily deactivate it by setting threshold of it’s log4j logger above DEBUG level. In case of web applications it is a good idea to wrap it in a Servlet filter that will call StopWatch.reset() at the beginning of a request and StopWatch.prettyPrint() at the end. You might also easily wrap all Spring managed beans with a simple AOP advice that would transparently track performance of individual methods etc. Here’s the code:

import java.util.*;
import org.apache.log4j.Logger;

public class StopWatch {

	private static final Logger log = Logger.getLogger(StopWatch.class);
	private static ThreadLocal<StopWatch> threadLocal = null;

	private long eventIndex = 0;
	private Stack<Event> events = new Stack<Event>();
	private Set<Event> eventsDone = new TreeSet<Event>(new EventComparator());

	private StopWatch() {
	}

	public static void reset() {
		if (log.isDebugEnabled() == false) return;
		getThreadLocal().set(new StopWatch());
	}

	public static void start(String eventName) {
		if (log.isDebugEnabled() == false) return;

		StopWatch stopWatch = getInstance();
		if (stopWatch != null) {
			stopWatch.startEvent(eventName);
		}
	}

	public static void stop() {
		if (log.isDebugEnabled() == false) return;

		StopWatch stopWatch = getInstance();
		if (stopWatch != null) {
			stopWatch.stopEvent();
		}
	}

	public static String prettyPrint() {
		if (log.isDebugEnabled() == false) return null;

		StopWatch stopWatch = getInstance();
		return (stopWatch != null) ? stopWatch.internalPrettyPrint() : null;
	}

	private void startEvent(String eventName) {
		eventIndex++;
		events.push(new Event(eventIndex, events.size(), eventName));
	}

	private void stopEvent() {
		if (events.size() == 0) return;

		Event event = events.pop();
		event.stop();
		eventsDone.add(event);
	}

	private String internalPrettyPrint() {
		StringBuilder sb = new StringBuilder();
		if  (events.size() != 0) {
			sb.append("WARNING: same events have not been marked as closed, ");
			sb.append("the summary might contain wrong data...\n");
		}
		for (Event event : eventsDone) {
			sb.append(getTabs(event.getDepth()))
			.append(event.getName()).append(" : ").append(event.getTime()).append(" ms\n");
		}

		return sb.toString();
	}

	private static String getTabs(int count) {
		StringBuilder sb = new StringBuilder(count);
		for (int i = 0; i < count; i++) {
			sb.append("\t");
		}
		return sb.toString();
	}

	private static StopWatch getInstance() {
		return getThreadLocal().get();
	}

	private static ThreadLocal<StopWatch> getThreadLocal() {
		if (threadLocal == null) {
			threadLocal = new ThreadLocal<StopWatch>();
		}
		return threadLocal;
	}

	private class Event {

		private long index;
		private int depth;
		private String name;
		private long startTime;
		private long time;

		public Event(long index, int depth, String name) {
			this.index = index;
			this.depth = depth;
			this.name = name;
			startTime = System.currentTimeMillis();
		}

		public void stop() {
			time = System.currentTimeMillis() - startTime;
		}

		public long getIndex()  { return index; }
		public int getDepth()   { return depth; }
		public String getName() { return name; }
		public long getTime()   { return time; }
	}

	private class EventComparator implements Comparator<Event> {
		public int compare(Event left, Event right) {
			if (left == null && right == null) return 0;
			if (left == null) return 1;
			if (right == null) return -1;
			return (int) (left.getIndex() - right.getIndex());
		}
	}
}

If you have suggestions on how to optimize or improve it in any way, I will be happy to hear from you!

At first sight this might not seem to be very useful, but if we bring into the picture security of web applications and especially cross-site scripting (XSS) vulnerabilities – things get interesting. One of the classical examples of XSS attack is the one in which a hacker manages to read user’s session identifier from a cookie and use it to access a resource⁽²⁾.

The most obvious way to remediate that would be to use HttpOnly attribute while setting the JSESSIONID cookie. Unfortunately this step is done by the application server itself and as on now most of them do not use HttpOnly⁽³⁾. What we might try to do is to rewrite the cookie after it has been created as shown here: http://keepitlocked.net/archive/2007/11/05/java-and-httponly.aspx.

But there’s also another way which you might consider.

We are going to use a Servlet filter to do following:

1. On the first request within a session we create a token – it should be a random value, just as JSESSIONID is. We store the token as a HttpOnly cookie and also as a session attribute.
2. On every subsequent request we compare whether the cookie token is equal to the value stored in session.
3. If cookie token is missing or they are different – we can assume that a third party is trying to impersonate the user by using stolen JSESSIONID.
4. In such case we can invalidate the whole session, inform an administrator about the attempt or even let the original user know that somebody has been trying to hack in.

Aside of giving us a chance to trace this particular type of abuse, this mechanism can also be used to deal with one more case. Some older servlet containers (that will remain nameless ) create session identifiers of a fixed length without providing any configuration option to make them longer. If your application happens to be audited – too short session identifier will most likely by treated as a vulnerability and required to be fixed.

By using the mechanism described above we can effectively control the length of session id, because from practical perspective the session identifier length will be equal to JSESSIONID length plus our hand-generated token length. In real life this approach has helped me to get approval of the ethical hacking team more than once.

A proof-of-concept code below, feel free to suggest changes.

import java.io.IOException;
import javax.servlet.*;
import javax.servlet.http.*;

public class HttpOnlyTokenFilter implements Filter {

    private static final String TOKEN_KEY = "HTTP_ONLY_TOKEN";

    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
        throws IOException, ServletException {

        validateToken((HttpServletRequest)request, (HttpServletResponse)response);
        chain.doFilter(request, response);
    }

    private void validateToken(HttpServletRequest request, HttpServletResponse response) {
        HttpSession session = request.getSession(true);
        String token = (String)session.getAttribute(TOKEN_KEY);
        if (token == null) {
            token = getRandomString();
            session.setAttribute(TOKEN_KEY, token);
            response.addHeader("Set-Cookie", TOKEN_KEY + "=" + token + ";httpOnly");
        }
        else {
            String cookieToken = getCookieValue(request.getCookies(), TOKEN_KEY);
            if (token.equals(cookieToken) == false) {
                session.invalidate();
            }
        }
    }

    private String getCookieValue(Cookie[] cookies, String cookieName) {
        if (cookies == null) return null;
        for (Cookie cookie : cookies) {
            if (cookieName.equals(cookie.getName())) return cookie.getValue();
        }
        return null;
    }

    private String getRandomString() {
        return String.valueOf(System.currentTimeMillis()); // TODO: replace with more random value. I mean it!
    }

    public void init(FilterConfig filterConfig) throws ServletException {}
    public void destroy() {}
}

⁽¹⁾It seems that soon all mayor browsers will support this feature as well, for example http://blogs.securiteam.com/index.php/archives/849
⁽²⁾ As in http://www.owasp.org/index.php/Session_hijacking_attack
⁽³⁾ Things are improving here as well: https://issues.apache.org/bugzilla/show_bug.cgi?id=44382