Roman Markunas (Romans Markuns) blog

Reducing JVM heap size, part 3: Manual memory management

Thu, 10 Sep 2020 00:00:00 +0100

In this article we will look at different ways to figure out and fix unnecessary memory consumption by JVM. This is useful for JVM applications that store huge amount of data in memory and can help avoid OutOfMemoryError if application data cannot be partitioned and scaled horizontally.

In part 3, we will take a look on more advanced techniques used to reduce JVM heap size, which involves manual memory management. This is the last part of series following part 2.

Space occupied by Java object

Our best result in part 2 was 481300 orders on a heap of 64MB, in an example 11. This is on average 139 bytes per object. At this point we optimised the encoding of data and extra memory required by data structures. Let’s look at the heap dump and see how much space every Order object is taking. Eclipse MAT is claiming it is 56 bytes, let’s see how this adds up:

Field name	Field type	Field size (bytes)
id	long	8
user	reference (byte[])	4
articleNr	int	4
count	int	4
pricePence	int	4
addressNumber	reference (String)	4
addressStreet	reference (String)	4
addressCity	reference (String)	4
addressRegion	reference (String)	4
addressPostCode	reference (String)	4
TOTAL		44

However, this is not a full picture as String, for example, is an indirection over byte[] and ads extra reference, byte and int field for each object.

Therefore, we are wasting:

5x (4 + 1 + 4) = 45 bytes for String objects we could be leaving off as byte[]
12 bytes of object header

Let’s try to tackle these two.

Pooling byte arrays

In previous part we were able to deduplicate String instances by using JVM’s pool. If we are to get rid of strings, we have to replace JVM pool with a custom one, handling byte[] deduplication for us.

However, I would like to take this a step further, as eventually we want to store Order object off heap. This means that we must avoid storing references on an Order. Object references in Java can be mutated by JVM during GC cycles and if we want to avoid serializing duplicate string values, we must replace byte[] references with keys to byte array pool.

In previous part we already tried to pool byte[] (see example 12), but did not succeed because of tree-structure wrapper overheads. To make this pooling worthwhile, we can use open-addressing implementation with the following interface:

public class PooledByteArrayMap {
    public int put(byte[] value);
    public byte @ImmutableByteArray [] get(int key);
    public void free(int key);
}

This data structure is efficient in deduplicating byte arrays with time complexity of a hash map and lookup/remove complexity of O(1). Source code is available here.

Using such pool, we are left without any references in Order object, but we can create temporary String objects during order processing:

public final class Order {

    private final long id;
    private final int user;
    private final int articleNr;
    private final int count;
    private final int pricePence;
    private final int addressNumber;
    private final int addressStreet;
    private final int addressCity;
    private final int addressRegion;
    private final int addressPostCode;

    @JsonCreator
    public Order(
            @JsonProperty("id") long id,
            @JsonProperty("user") byte[] user,
            // other fields omitted
            @JacksonInject PooledByteArrayMap byteArrayPool
    ) {
        this.id = id;
        this.user = byteArrayPool.put(user);
        // other fields omitted
    }

    // other getters omitted

    public String getAddressPostCode(PooledByteArrayMap byteArrayPool) {
        return restoreString(addressPostCode, byteArrayPool);
    }    

    @SuppressWarnings("byte.array.weakening")
    private String restoreString(int key, PooledByteArrayMap byteArrayPool) {
        return new String(byteArrayPool.get(key), StandardCharsets.US_ASCII);
    }
}

With this approach we can fit 576700 orders (+20%) into memory, which is about 116 bytes (-23) per object. We were not able to get all 45 bytes worth of gain by ditching String due to some overheads of a pool, but it is an improvement nonetheless.

This change has 2 unintended consequences which are discussed below.

Replacing object encapsulation guarantees with compile checks

One of key features of Java standard library String class is its immutability. By removing it altogether we are not protected from accidentally changing byte[] content, which would alter results of business transactions and compromise integrity of pool. The cost of safety was too high to bear at runtime, however we can move that cost to compile time.

Since Java 8, javac allows adding custom type checkers which would be run as part of code compilation. For example, this allows developers to annotate certain variables as not nullable and avoid pesky null checks throughout the code. Similarly, we can declare all byte[] instances, returned by the pool immutable using annotation:

public byte @ImmutableByteArray [] get(int key);

I’ve decided to implement such type as a pair of annotation/checker within Checker Framework. This ensures that all attempts to modify byte[] returned by pool or to downcast them to unannotated version would be flagged as compiler error. Checker source can be found here.

This approach gives back confidence, without incurring run time costs. Similar approaches can be used to replace type-safe wrapper objects in Java (AKA tiny-types) with primitives to avoid runtime costs and maintain type-safety.

Array utilisation

For heap sizes like in this scenario there is another important consideration to take into account. Running previous example 14 with GC logging enabled we can see heap statistics at point of failure:

[33.920s][info][gc,heap,exit   ] Heap
[33.920s][info][gc,heap,exit   ]  garbage-first heap   total 65536K, used 61378K [0x00000000fc000000, 0x0000000100000000)
[33.920s][info][gc,heap,exit   ]   region size 1024K, 1 young (1024K), 0 survivors (0K)
[33.920s][info][gc,heap,exit   ]  Metaspace       used 12518K, capacity 12795K, committed 13056K, reserved 1060864K
[33.920s][info][gc,heap,exit   ]   class space    used 1139K, capacity 1197K, committed 1280K, reserved 1048576K

From this we can observe that about 4MB of heap space was not utilised. One reason for this is humongous object allocation. Every time JVM with G1 GC tries to allocate an object with size larger than region size (1MB in this case) it will combine multiple regions, which then will be used solely by that humongous object. A few megabytes wasted on a multiple gigabyte heap may be unnoticed, however for 64MB heap it makes a considerable difference.

Other thing to consider is array list utilisation. Some space can be wasted, if the list get resized too early and underlying array grows prematurely exacerbating issues with heap fragmentation. In our case, array lists get resized at 50% capacity (default for Agrona collections).

To counter all these effects, I’ve tried following improvements (see example 15):

changing load factor for collections to 85%
custom byte array pool would split underlying arrays into smaller chunks and allocate an extra chunk on resize
do not use G1 to reduce memory fragmentation due to humongous allocations. This will not alleviate fragmentation completely as every GC has trouble defragmenting memory with object sizes approaching 10% of total heap. I’ve used serial GC, because for small heaps CMS and throughput collectors can take very long time to fail.

This allows fitting 686200 orders (+19%) into memory, which is about 98 bytes (-18) per object.

SLAB allocation

Since Order object does not hold any object references we can manage allocation ourselves and avoid JVM object header overheads. To accomplish that, we need some sort of allocator to store/retrieve objects for us and provide an abstraction over plain memory access. SLAB is one of simpler allocators that can achieve that. I’ve used the simplest form of SLAB, similar to Linux SLOB allocator (see source), with a following interface:

public class OrderSlabAllocator {

    public int allocate();
    public OrderView get(int key);
    public void free(int key);
}

Using this store we no longer need to store a reference to Order object, but only an object key within allocator. In this case key would correspond to some contiguous chuck of memory. However, we still would like a simplicity of using familiar getter/setter API, instead of manipulating bits in memory. To do this we can utilize a flyweight pattern:

public final class OrderView {

    private final PooledByteArrayMap byteArrayPool;

    private ByteBuffer buffer;
    private int startPosition;


    OrderView(PooledByteArrayMap byteArrayPool) {
        this.byteArrayPool = byteArrayPool;
    }

    OrderView wrap(ByteBuffer buffer, int startPosition) {
        this.buffer = buffer;
        this.startPosition = startPosition;
        return this;
    }

    public OrderView set(
            long id,
            byte[] user,
            int articleNr,
            int count,
            int pricePence,
            String addressNumber,
            String addressStreet,
            String addressCity,
            String addressRegion,
            String addressPostCode
    ) {
        buffer.putLong(startPosition + ID_OFFSET, id);
        buffer.putInt(startPosition + USER_OFFSET, byteArrayPool.put(user));
        // other fields write omitted
        buffer.putInt(startPosition + ADDRESSNUMBER_OFFSET, internString(addressNumber));
        return this;
    }


    public long getId() {
        return buffer.getLong(startPosition + ID_OFFSET);
    }

    public byte @ImmutableByteArray [] getUser() {
        return byteArrayPool.get(getUserPoolKey());
    }

    public String getAddressNumber() {
        int addressNumber = buffer.getInt(startPosition + ADDRESSNUMBER_OFFSET);
        return restoreString(addressNumber);
    }

    // the rest of getters are omitted

    private int internString(String str) {
        return byteArrayPool.put(str.getBytes(StandardCharsets.US_ASCII));
    }

    @SuppressWarnings("byte.array.weakening")
    private String restoreString(int key) {
        return new String(byteArrayPool.get(key), StandardCharsets.US_ASCII);
    }
}

We only need single instance of OrderView to manipulate orders. When we ask allocator for an order, it would wrap this view around correct memory location and allow us to use order as if it was bog-standard POJO.

This technique is one step away from doing off-heap storage, where byte buffers are replaced with appropriate DirectBuffer or MappedByteBuffer. Compression technique can also be more beneficial when applied after this step, because having multiple orders compressed together is more likely to contain repetitive data allowing for better compression.

This storage pattern also has an added performance benefit, as memory access is much more cache-friendly.

This approach lets us fit 814900 orders (+19%), which is 82 bytes (-16) per object.

Conclusion

Working through multiple optimization techniques we were able to reduce memory consumption by 300% (4 times). This is quite good result for data, which is not highly compressible. Of course, there is an inherent limit on what can be fit in given unit of memory. This limit is defined by the amount of data represented using minimal viable types, and can be reduced only by data sharing between objects or compression.

For this particular example, we have not reached that limit yet. Higher data density can be achieved by changing key types from int to 3 bytes (we know we cannot have more than 2^23 object fit into memory anyway) and compressing SLABs.

Reducing JVM heap size, part 2: Optimizing

Wed, 01 Jul 2020 00:00:00 +0100

In part 2, we will take a look on couple of techniques used to reduce JVM heap size. The techniques listed in the order of least invasive to most, so that optimization does just enough for application to work and keeps it as maintainable as possible. The example application can be found in part 1 or in example source code.

Memory efficient data structures

As mentioned in part 1 we could fit around 204500 orders/54600 users into 64M heap. Let’s take a look on how we can improve this.

The least intrusive change is usually changing data structures that hold objects. These usually conform to some of Java collection interfaces and changing them would not introduce much change in the code. Let’s see how much memory we can save by doing so.

The first obvious candidate would be ArrayList as it’s no arg constructor would create a list which will have capacity of 10 after first insertion. However, we have around 200K orders across 50K users, which brings the average number of orders per list to 4. We can be aggressive and set that size to 1 initially. That gives us 209000 orders/54900 users - an easy win, but only 2%.

Second optimization has to do with java HashMap, which uses simple chaining implementation. That is memory efficient only if map key collision rate is low. In our case both order and user ids collision rate is about 18% (see this test), so there might be potential here. Maps that use open addressing implementation remain memory efficient even with high collision rates. Any open addressing implementation should do, as collision resolution strategy (probing, Robin-Hood, cuckoo, etc) should not significantly impact memory footprint.

I’ll use Agrona Object2ObjectHashMap to test this. Agrona’s implementation also does not create extra Node objects, as it stores keys and values as even and odd indexes in the same array. Changing map implementation (example 8) gives us 231500 orders/56000 users, which is significant 11% increase.

Using minimal types to store data

Let’s take a look at Order objects themselves.

First step is to look at the structure of stored values. We want them to have as flat structure as possible, minimizing extra object header overhead. In our case we would just get rid of a boxed type. This brings us to 245900 orders/56600 users, which is another 6%. Quite significant increase for this little work.

Now when the low hanging fruit has been plucked, we should take a closer look at Order and evaluate if we represent data with appropriate types.

In or case, we have another field which is secretly boxed, which is Order#id. At the moment, this field is just a number, but wrapped as String. Most times there is no need for id field to be a string, as we can represent enough unique identifiers with a simple number (or a tuple of numbers) and have a Long keys in a map:

public final class Order {

    private final long id;
    private final String user;
    private final int articleNr;
    private final int count;
    private final int price;
    private final String address;

    // rest of class omitted for clarity
}

Map<Long, Order> ordersById = new Object2ObjectHashMap<>();

By doing this, we can fit 276100 order/57900 users, and that’s another 12% increase. However, we are now forced to waste some memory for autoboxing long, which we can fix by using primitive collection. There are ready libraries to do this, for example Trove collections. Here I’ll use Agrona, since I’ve already used it in example before:

Long2ObjectHashMap<Order> ordersById = new Long2ObjectHashMap<>();

That’s 288300 orders/58300 users, which is extra 4%, and the final code can be observed in example 9.

Data tokenization and shared object pool

At this point most of the memory would be occupied by address String. The problem here is that addresses contain lots of repeating information: cities, post codes, street numbers, names and surnames, etc. What we can try to do, is to break address apart into separate components (tokens) and share repeating instances between Order values. JVM even has built in String pool for us to use!

I’ve decided to break up address like this:

public final class Order {

    private final long id;
    private final String user;
    private final int articleNr;
    private final int count;
    private final int pricePence;
    private final String addressNumber;
    private final String addressStreet;
    private final String addressCity;
    private final String addressRegion;
    private final String addressPostCode;

    // rest of class omitted for clarity
}

Without pooling, we can only fit 144200 orders/50000 users into memory, which again proves that object header overhead is very significant. It might seem like a step back, but let’s turn String pooling on in Order constructor (example 10):

@JsonCreator
public Order(
        // some fields omitted
        @JsonProperty("addressNumber") String addressNumber,
        @JsonProperty("addressStreet") String addressStreet,
        @JsonProperty("addressCity") String addressCity,
        @JsonProperty("addressRegion") String addressRegion,
        @JsonProperty("addressPostCode") String addressPostCode) {
    // some fields omitted
    this.addressNumber = addressNumber.intern();
    this.addressStreet = addressStreet.intern();
    this.addressCity = addressCity.intern();
    this.addressRegion = addressRegion.intern();
    this.addressPostCode = addressPostCode.intern();
}

which lets us fit 386000 orders/61000 users into memory (+34%)! JVMs with version 7 and higher do not allocate interned String objects in native memory, therefore this gain is fair comparison (this can be confirmed by running JVM with flags -XX:+UnlockDiagnosticVMOptions, -XX:NativeMemoryTracking=summary, -XX:+PrintNMTStatistics and inspecting memory usage).

As far as I know, String pool is the only JVM pool which is usable to applications (in contrast, Integer pool goes only up to number 128). However, it shouldn’t be hard to pool any user defined object. Usability of this method depends on how much of data is repeating and how it could be split. Developers, considering this technique, should bear in mind object overhead of internal pool implementation (e.g. be careful with tree-like collections which allocate extra objects for every node).

Data encoding

Compression approach can save some memory at the expense of processing power and code maintainability as switching form JDK types to custom ones limits the usage of library methods.

Since we only use capital letters and digits for ids, we can encode strings using base 36 encoding. Base 36 values would take only 6 bits each, instead of usual 8 bits (16 bits before Java 9) per character, therefore it is expected to save about 25% of memory (60% before Java9):

public Order(
            @JsonProperty("id") long id,
            @JsonProperty("user") byte[] user,
            // rest omitted

and we only need to store reference to string as a map key (byte[] are not suitable as map keys as they do not override hashCode/equals properly and are mutable):

ordersByUser
        .computeIfAbsent(
                new Base36String(order.getUser()), 
                key -> new ArrayList<>(1)
        )
        .add(order);

More compact encoding (example 11) and not storing extra reference on Order objects allows us to stuff JVM with 481300 orders/62800 users (+24%).

Ditching unnecessary references

Now we can try to get rid of extra String object references and store only byte[]. If we ever need a String object later in code we can allocate a short-lived object for that operation or use a view wrapper. This is somewhat risky, because arrays in Java are not immutable, but let’s see how much memory we can save that way.

Simple type replacement, however, would not work here as our strings are pooled objects. Therefore, we need to roll out our own pool for byte[]. This is a bit tricky in Java, cause Java arrays do not properly implement hashCode or equals, but we can leverage array order and TreeSet:

public class TreeBasedByteArrayPool implements ByteArrayPool {

    private final NavigableSet<byte[]> values = new TreeSet<>(Arrays::compare);


    @Override
    public byte[] intern(byte[] in) {
        boolean newValue = values.add(in);
        if (newValue) {
            return in;
        }
        else {
            return values.floor(in);
        }
    }
}

and inject pool during deserialization like so:

@JsonCreator
public Order(
        @JsonProperty("id") long id,
        // more fields here ...
        @JacksonInject ByteArrayPool byteArrayPool
)

ObjectMapper mapper = new ObjectMapper();
mapper.setInjectableValues(new InjectableValues.Std().addValue(
        ByteArrayPool.class, 
        new TreeBasedByteArrayPool()
));

This approach (example 12) doesn’t help us much, because by getting rid of some references in Order object, we’ve created many more references within TreeSet, and thus for this use-case we can actually fit fewer objects in memory. Even linear-time array-based implementation does not reduce memory footprint much, which means, that for our data structure this tree-like pool implementation is not sufficient. More elaborate pool implementation, based on open-addressing, is used in further example 14 of part 3.

Object compression

Now what if we could compress the whole Order object? In our case we must only compress fields which are not shared between Order instances (non-address fields are transient), hence we end up with something like this (example 13):

public class OrderCompressed {

    private final byte[] compressedPart;
    // shared fields omitted
    // constructor omitted

    public static OrderCompressed compress(Order order) {
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        try (
                GZIPOutputStream gzipOut = new GZIPOutputStream(out);
                ObjectOutputStream objectOut = new ObjectOutputStream(gzipOut);
        ) {
            objectOut.writeObject(order);
        }
        catch (IOException e) {
            throw new RuntimeException("Unable to serialize " + order, e);
        }
        byte[] bytes = out.toByteArray();
        return new OrderCompressed(
                bytes,
                order.getAddressNumber(),
                order.getAddressStreet(),
                order.getAddressCity(),
                order.getAddressRegion(),
                order.getAddressPostCode()
        );
    }
}

In our case this also does not give any memory reduction, as we are only compressing couple of primitive fields. This technique is much more useful where it is impossible to share much data between retained objects or object does not contain references. This technique can prove more useful in conjunction with manual memory management described in next part.

Conclusion

Using appropriate data structures, variable types, compression and pooling we have reduced memory consumption by 135% or 2.35 times (fitting 481300 orders instead of 204500).

More memory usage reduction can be achieved by bulk compressing larger chunks of data together or moving off-heap, which is covered in part 3. These however impact code and architecture quite a bit and must be considered along with data sharding approaches.

Reducing JVM heap size, part 1: Profiling

Wed, 01 Jul 2020 00:00:00 +0100

In this series we will look at different ways to figure out and fix unnecessary memory consumption by JVM. This is useful for JVM applications that store huge amount of data in memory and can help avoid OutOfMemoryError if application data cannot be partitioned and scaled horizontally.

In part 1, let’s take a look on how to identify which objects are responsible for increased heap size. In part 2, I’ll list a few optimization techniques that should help reduce heap size.

We will not look at memory leak detection in this post, however most profiling methods in this article will be useful in finding those too. We will also not focus on garbage-free approach to Java code. Let me know if you’re interested in these topics as well.

Source code of examples can be found here.

Example application

This corresponds to example 1.

To emulate application with long-term memory store, we will launch separate JVM and we are going to feed it orders as such:

public final class Order {

    private final String id;
    private final String user;
    private final Integer articleNr;
    private final int count;
    private final int price;
    private final String address;

    // rest of class omitted for clarity
}

To compare runs of different implementations instances of Order are created deterministically by seeding Random with the same seed.

To represent real-life data scenario we will take Alaska, USA address dataset collected by OpenAddresses, courtesy of Municipality of Anchorage.

Orders are sent via stdin pipe, read by Scanner and stored in example application in 2 maps, keyed by id and user:

String received = scanner.nextLine();
Order order = mapper.readValue(received, Order.class);
ordersById.put(order.getId(), order);
ordersByUser
        .computeIfAbsent(order.getUser(), key -> new ArrayList<>())
        .add(order);

Running our memory hungry application in JVM with -Xmx64m` we get:

Total orders: 204785
Total users: 54643
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.Reader.read(Reader.java:188)
	at java.base/java.util.Scanner.readInput(Scanner.java:882)
	at java.base/java.util.Scanner.findWithinHorizon(Scanner.java:1796)
	at java.base/java.util.Scanner.hasNextLine(Scanner.java:1610)
	at com.romanmarkunas.blog.memory.example1.OrderStoreMain.main(OrderStoreMain.java:19)

Let’s pretend we don’t know what is the most memory hungry part of application and see how we can find out biggest memory consumers within JVM.

Profiling - memory release pattern

Much of the profiling we do in this section relies on production-like data for real system. This is why it is highly advised to have most of mentioned JVM options enabled proactively for production system to properly monitor memory consumption.

First we need to identify what is the source of OutOfMemoryError. Potentially we could have a current thread stack allocating and holding too many objects as part of it’s immediate operation (e.g. loading a huge file into memory in one go), so there might be problem with sizing JVM memory or algorithm. Alternatively, if we see a “sawtooth” pattern we could conclude that memory is being held long term. This corresponds to example 2.

Running with -Xlog:gc (or -verbose:gc prior to Java 9), would give us following (-XX:+PrintGCDetails omitted for clarity, run java -Xlog:help for more options on logging post Java 8):

[0.008s][info][gc] Using G1
[0.455s][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 24M->3M(64M) 5.136ms
[0.675s][info][gc] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 38M->7M(64M) 18.182ms
...
[2.021s][info][gc] GC(77) To-space exhausted
[2.021s][info][gc] GC(77) Pause Young (Normal) (G1 Evacuation Pause) 62M->61M(64M) 2.543ms
[2.027s][info][gc] GC(78) To-space exhausted
[2.027s][info][gc] GC(78) Pause Young (Normal) (G1 Evacuation Pause) 63M->63M(64M) 3.311ms
[2.102s][info][gc] GC(79) Pause Full (G1 Evacuation Pause) 63M->54M(64M) 74.398ms
...
[7.532s][info][gc] GC(171) Pause Full (G1 Evacuation Pause) 63M->63M(64M) 119.835ms
[7.532s][info][gc] GC(172) Pause Young (Normal) (G1 Evacuation Pause) 63M->63M(64M) 0.231ms
[7.655s][info][gc] GC(173) Pause Full (G1 Evacuation Pause) 63M->62M(64M) 122.730ms
// OutOfMemoryError here

Just as interesting side note - OutOfMemoryError happens while heap has little space left. Even though G1 is a compacting collector, slight memory fragmentation occurs (For example, each object from reclaimed region is too big to fit in left free space of almost full regions. Another case would be when, any region is taken by humongous object - more reading here).

Observing used heap size after each collection (number just after ->), we can see that it is always growing, creating infamous saw-tooth pattern. We can also observe (with -Xlog:gc*) that G1 garbage collector shrunk young generation to single region (1024K) by the time of crash:

[4.714s][info][gc,heap,exit   ] Heap
[4.714s][info][gc,heap,exit   ]  garbage-first heap   total 65536K, used 64505K [0x00000000fc000000, 0x0000000100000000)
[4.714s][info][gc,heap,exit   ]   region size 1024K, 1 young (1024K), 0 survivors (0K)

This suggests we have either too much data or memory leak. Next, let’s find out what exactly fills the memory.

Profiling - heap dumps

Let’s start with easiest approach, which is option -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof. This will give use heap dump in binary format. Now to make sense of it, we need to open with any useful dump analysis tool. I do not recommend VisualVM, because it’s very bad in guessing object sizes, and for our particular example it would show retained sizes bigger than heap itself. It also does not include thread frames in dominator lists, which often is helpful. For this example I’ll use Eclipse MAT, which is a free, standalone, feature-full and precise memory analyser tool.

Once we open heap dump (example 3), it’s quite obvious to figure out where most JVM memory is consumed:

Obvious offender

If we open dominator tree and sort by retained heap, we will see why it is the case:

Dominating Thread

You may wonder why Order objects in dominator tree are children of Thread instead of HashMap and why retained heap of both HashMap’s is much smaller than total retained heap. The reason is that all Order objects are retained by both maps, so if any single map becomes unreachable, none of Order objects would be collected. However, if Thread becomes unreachable, both HashMap and in turn all Order will be eligible for garbage collection.

Retained heap of the object is total memory that would be reclaimed by GC if this object would become unreachable.

This is exactly reason why dominator trees are not useful to figure out memory consumption in real-life application. Big data structures like caches are often shared between components and referenced via multiple GC roots, therefore they do not appear in dominator trees. To demonstrate this we can put our order map as a static variable (example 4):

public class TwoGCRootsOrderStoreMain {

    private static Map<String, Order> ordersById = new HashMap<>();

    public static void main(String[] args) {
        Map<String, Order> ordersById = TwoGCRootsOrderStoreMain.ordersById;
        // rest of code omitted
    }
}

and the dominator tree of that would look like:

No clear winner

which is not immediately helpful - top dominators take 25% of memory, while we clearly filled whole heap. In cases like above, histograms are more useful, so lets create one and sort items by retained heap size:

Object histogram

Here we can see a huge amount of Order objects and, assuming it is our normal production scenario, we have found our offender.

Histograms can be useful in finding memory/resource leaks in conjunction with other metric collection tools for your system. For example, if you publish count of current active TCP connections via metrics interface, but then you find out that number of Socket objects in heap is much bigger, it is an obvious resource leak.

For a typical heap it is very likely that top objects would be byte[], String, HashMap$Node, Object[] and ConcurrentHashMap$Segment. And in bigger application it’s typically not obvious what is holding all these objects. To find out right click on desired class object and select “Immediate dominators”. Our example for String would give:

String dominators

which tells us, that most String instances are retained through Order objects or a GC root, which is main Thread. Also in histogram view: right-click on String -> “Merge shortest paths to GC roots”, would give:

String retain tree up to GC roots

Which exact objects retain String

which tell us that String objects are specifically retained as:

Order object fields user and address, which in turn are HashMap$Node values
HashMap$Node keys

Again Order#id is not retaining strings, as collecting order would not result in collection on id String instance, because it is also retained as HashMap$Node key. Collecting the whole node, however would cause id to be collected, as both key and value would no longer be reachable.

To obtain heap dump of live process, the recommended tool since Java 9 release is jmap which is shipped together with JDK, for example jmap -dump:live,format=b,file=dump.hprof <java-process-pid>. Alternatively, if JVM is launched with PrintClassHistogram option kill -QUIT <java-process-pid> will also do it. However, the latter option overrides default behavior of printing thread dump on SIGQUIT).

Profiling - online profilers

Analyzing heap dump is great, but not always possible as it causes stop-the-world pause, therefore not applicable when monitoring production systems (heap dumps can induce very long pauses). To avoid this we can use online profilers, that collect sample data during runtime. Of course, we don’t get full picture using such method, but it still can give some clues on how application works and narrow the suspect circle.

Let’s take a look at JDK flight recorder, which is free feature of JDKs as of version 11. To get this right we will need to tweak quite a few configuration options to enable continuous recording so that we can periodically dump sample data (another option would be to start recording periodically using jcmd <PID> JFR.start <all the same otions> command):

-XX:+FlightRecorder - this enables JDK flight recorder
-XX:FlightRecorderOptions:
memorysize=5m, - be aware of additional buffer that is required to collect sampling data, with size set here. Enabling JFR would also somewhat expand metaspace and use some extra heap.
stackdepth=32 - maximum stack depth to walk. Tuning this can help limit sampling overhead, cause some part of the stack might not be interesting, e.g. framewrok/servlet container dispatching
-XX:StartFlightRecording: - starts a recording as part of JVM launch
name=SampleRecording, - assigning name can help to dump the correct recording using jcmd <pid> JFR.dump name=SampleRecording
settings=default, - path to setting file, which instructs JFR what kind of events we care about. We will have to make our own to profile memory, see below, however default is very low overhead and can be safely run in production continuously
disk=false - do not write recordings immediately to disk. This makes sense for continuous recording when we want to periodically dump ourselves
dumponexit=true, - dump sampling data if JVM crashes unexpectedly
maxage=1h - maximum length of sampling data to be kept in buffer
-Xlog:jfr* - log out JFR events for the curious

2 settings files, that come by default with jdk do not collect memory data. Let’s tweak them using JDK Mission Control app, which we also will need to analyze recording data later. You may want to download build from OpenJDK, if you, like me have problems with Live Object histogram rendering. To do so, open Window -> Flight Recording Template Manager and duplicate <jdk directory>/lib/jfr/default.jfc. Then change “Memory Leak Detection” to “Object types + Allocation Stack Traces + Path to GC Root”. Export this file and specify its location using settings option above. Because we do not alter memory profiling options, the extra overhead should not be too big, however, be careful and measure if you plan to use this in production.

Once we have a profile ready (see example 5 for full option list) we can drag-and-drop it into JMC and inspect. We are interested primarily in “Live Objects” section (if you can’t see histogram with live objects on the top, try pulling top of table down):

Sampled live objects

This is a list of sampled objects for a period of time encircled with red. The total count will be much less than actual allocation count, but statistically it would be a representative sample of objects allocated and still alive. The samples are taken from TLAB every time it is filled and JVM assigns another TLAB to a thread. The exact mechanism how samples are selected from TLAB is described here - for general sampling approach and here - for Old Object Sample event.

What we can tell from that list is that all these object were still alive at the point when record was taken (JVM crash in our case), therefore some of these objects are candidates for memory leak/memory optimisation. We can find out which part of the code retain them by tracing allocation StackTrace in the bottom section and looking through code where these new objects were stored afterwards. In our case we can see that lots of String and HashMap internal objects are allocated and retained by Order objects.

One can notice that overall number of samples is quite low. The reason is that samples are chosen once TLAB is full and in this case TLAB size is relatively big to overall heap size (64 megabytes), hence small amount of TLABs and samples. By forcing smaller TLAB we can get more samples (example 7).

Another useful section can be “TLAB Allocations”, especially if you think you allocate too many big objects, that do not fit TLAB. To have that section populated, we must change “Memory Profiling” configuration to “Object Allocation and Promotion”. Or just use memory-all.jfc provided with example code.

Overall we can reconstruct quite similar picture to what we had when looking at the heap dump. It is bit more involved as we deal with sampled data and need more manual tracing, however we do not pay penalty of increased latency.

Intermission

In this part we have covered a few ways to determine what data is taking heap space. We have not covered probing here, as probing gives diminishing returns when we talk about heap occupation. In the second part we will take a look at what can be done to reduce heap usage by our sample program.

Jackson mix-ins

Wed, 03 Jun 2020 00:00:00 +0100

Jackson mix-ins are crucial to clearly separate application layers according to the onion architecture (popularised further by Rob Martin), without code duplication and boilerplate. According to onion architecture marshalling would sit in infrastructure level and domain objects should not be dependent on messaging infrastructure.

Another important use case would be if you do not control source code for serialised objects, but rather import them as library. In this case it is impossible to annotate imported class.

Source code of examples can be found here.

Example that would just work™

Let’s say our library/domain class looks like this:

public class JacksonFriendlyZoo {

    private int giraffeCount;
    private boolean open;


    public JacksonFriendlyZoo() {
        // for deserialization, don't use me!
    }

    public JacksonFriendlyZoo(int giraffeCount, boolean open) {
        this.giraffeCount = giraffeCount;
        this.open = open;
    }


    public int getGiraffeCount() {
        return giraffeCount;
    }

    public boolean isOpen() {
        return open;
    }
}

Then both (de-)serialization would just work with Jackson. However we should not alter domain object to make infrastructure work. So let’s see how to work around common cases below.

Immutable objects

Let’s start by removing no arg constructor because it leaks deserialization concern into domain code:

public class ImmutableZoo {

    private final int giraffeCount;
    private final boolean open;


    public ImmutableZoo(int giraffeCount, boolean open) {
        this.giraffeCount = giraffeCount;
        this.open = open;
    }


    public int getGiraffeCount() {
        return giraffeCount;
    }

    public boolean isOpen() {
        return open;
    }
}

This serializes well, but attempting to deserialize this would cause:

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.romanmarkunas.blog.jackson.ImmutableZoo` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
 at [Source: (String)"{"giraffeCount":5,"open":true}"; line: 1, column: 2]

Let’s add a creator constructor using mix-in:

public abstract class ImmutableZooMixIn {

    @JsonCreator
    public ImmutableZooMixIn(
            @JsonProperty("giraffeCount") int giraffeCount,
            @JsonProperty("open") boolean open
    ) {}
}

And configure object mapper like so:

@Test
void canDeserializeWithoutDefaultConstructorButWithMixIn() {
    ObjectMapper objectMapper 
            = new ObjectMapper()
            .addMixIn(ImmutableZoo.class, ImmutableZooMixIn.class);
    String jsonIn = "{\"giraffeCount\":5,\"open\":true}";

    ImmutableZoo zoo = objectMapper.readValue(jsonIn, ImmutableZoo.class);
}

Improving code navigatability

Now you, like me, might be tempted to use creator factory method to leverage IDE code navigation features and have easier time to find out all places where object might be constructed, like so:

public abstract class ImmutableZooMixInFail {

    @JsonCreator
    public static ImmutableZoo create(
            @JsonProperty("giraffeCount") int giraffeCount,
            @JsonProperty("open") boolean open
    ) {
        return new ImmutableZoo(giraffeCount, open);
    }
}

however this would still fail with:

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.romanmarkunas.blog.jackson.ImmutableZoo` (no Creators, like default constructor, exist)... blah, blah

This uncovers one important insight on how mix-ins work in Jackson - it does not look at mix-in class to find creator, but rather it sees the target class with added corresponding annotations from mix-in class. In other words, if target class does not have static factory method with exactly same signature as mix-in’s annotated static factory, the mix-in would not have any effect.

That means, if we don’t have factory on domain/library object, we can make code more navigatable only for non-final classes:

public abstract class ImmutableZooMixInNavigatable extends ImmutableZoo {

    @JsonCreator
    public ImmutableZooMixInNavigatable(
            @JsonProperty("giraffeCount") int giraffeCount,
            @JsonProperty("open") boolean open
    ) {
        super(giraffeCount, open);
    }
}

In any case the link between target and mix-in classes can be discovered when checking class usage and finding ObjectMapper configuration.

Methods not following Java property convention

By default Jackson looks for methods in the form of Java property to serialize objects.

For example, to serialize field giraffeCount you must have method getGiraffeCount(). So if our domain/library class looks like this:

public class NotFollowingJavaPropertyConventionZoo {

    private final int giraffeCount;
    private final boolean open;


    public NotFollowingJavaPropertyConventionZoo(
            int giraffeCount,
            boolean open
    ) {
        this.giraffeCount = giraffeCount;
        this.open = open;
    }


    public int giraffeCount() {
        return giraffeCount;
    }

    public boolean open() {
        return open;
    }
}

it will fail to serialize with:

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class com.romanmarkunas.blog.jackson.NotFollowingJavaPropertyConventionZoo and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS)

This could be solved with mapper configuration:

new ObjectMapper().setVisibility(
        PropertyAccessor.FIELD, 
        JsonAutoDetect.Visibility.ANY
);

However this would alter serialization for all classes, in potentially breaking way. This can be solved on a per-class basis using mix-ins:

public abstract class NotFollowingJavaPropertyConventionZooMixIn {

    @JsonProperty("giraffeCount")
    public abstract int giraffeCount();

    @JsonProperty("open")
    public abstract boolean open();
}

ObjectMapper objectMapper = new ObjectMapper().addMixIn(
        NotFollowingJavaPropertyConventionZoo.class, 
        NotFollowingJavaPropertyConventionZooMixIn.class
);

NotFollowingJavaPropertyConventionZoo zoo 
        = new NotFollowingJavaPropertyConventionZoo(42, false);
objectMapper.writeValueAsString(zoo);

Fields that should not be deserialized

Another interesting example would be an object that has some properties that we don’t want to serialize, for example, a cached field:

public class HasPropertiesToIgnoreZoo {

    private final int giraffeCount;
    private final int sealCount;
    private final int totalCountCache;


    public HasPropertiesToIgnoreZoo(int giraffeCount, int sealCount) {
        this.giraffeCount = giraffeCount;
        this.sealCount = sealCount;
        this.totalCountCache = giraffeCount + sealCount;
    }


    public int getGiraffeCount() {
        return giraffeCount;
    }

    public int getSealCount() {
        return sealCount;
    }

    public int getTotalCount() {
        return totalCountCache;
    }
}

We can force total count to be ignored using following mix-in:

public abstract class HasPropertiesToIgnoreZooMixIn {

    @JsonIgnore
    public abstract int getTotalCount();
}

Instead of conclusion

Hope this helps and leave comment below if you want me to cover specific topic, thanks!

Getting started with Dropwizard

Thu, 15 Nov 2018 00:00:00 +0000

Dropwizard is a framework for rapid application development. Dropwizard pulls and glues together Jetty application server, Jersey REST framework, Jackson for serialization/deserialization and a bunch of other libraries, so you have everything at hand to develop full-featured Java-based RESTful web-applications.

This tutorial will go through core Dropwizard components and how they are assembled together to have working Dropwizard application within munites.

Ready sample application can be found here.

Pulling necessary dependencies

For this section I’ll be using Gradle as dependency manager, since Dropwizard website already has examples using Maven.

For simple app we need only one dependency + junit for unit tests:

dependencies {
    compile "io.dropwizard:dropwizard-core:$vDropwizard"

    testCompile "junit:junit:4.12"
    testCompile "io.dropwizard:dropwizard-testing:$vDropwizard"
}

Dropwizard testing package contains junit rules to create endpoints or whole embedded application instance within tests.

Application entry point

Since Dropwizard’s artifact is executable JAR, we need to define application entry point as part of our application class:

public class HelloApplication extends Application<Configuration> {

    public static void main(String[] args) throws Exception {
        new HelloApplication().run(args);
    }
    
    @Override
    public void run(Configuration configuration, Environment environment) {}
}

Main method is not necessarily to be located here, but it’s usually done so by convention. Now all we have to do in main method is to call for #run(args) which will startup application container, parse configuration and do all other lovely stuff we’ll discuss further in this article.

Application container

Dropwzard applications are served by embedded Jetty container. Most of configuring ang gluing Jetty together with other Dropwizard parts is already done (if you are curious how Jetty is launched within Dropwizard, take a look at io.dropwizard.cli.ServerCommand#run method - this is server entry point after successful input and configuration initialization). What we need to do, is to configure some application-specific server parameters in configuration file:

server:
  type: simple
  applicationContextPath: /

For this example I used simple Jetty server factory, which creates server with reasonable default settings and public/admin ports being 8080/8081. The only setting specified here is applicationContextPath, because default value is /applciation/ and I don’t want all endpoints to be prefixed with that.

At startup Dropwizard will parse this yaml file and deserialize it into Java factories that are used to create objects representing different components.

In case you wonder what type: simple means in this configuration - this is value of @JsonTypeName annotation. That way Jackson will know concrete ServerFactory implementation class that will need to be deserialized from this configuration.

Running

So once we have application class and configuration in place we can run our app! Let’s assume for now that we will run it from IDE as we’ll talk a bit more about packaging later.

To run app main method:

java ApplicationClass server path/to/config.yaml

If you follow supplied code this will result in:

java com.romanmarkunas.dwtutorial1.HelloApplication server config/config.yml

The successful launch will manifest with:

org.eclipse.jetty.server.Server: Started @3908ms

However scrolling up we can also see:

The following paths were found for the configured resources: NONE

this means we don’t have any endpoints defined yet. So currently we have a full-fledged server running for no reason. Let’s fix that!

Defining endpoint

For training purposes we will an endpoint that will return current server time and how many times this endpoint is invoked. Since Java methods can only return 1 value, we will need to create a POJO with these 2:

public class TimeAndInvocations {

    // fields and constructor omitted
    
    @JsonProperty
    public LocalTime getTime() {
        return time;
    }

    @JsonProperty
    public int getCurrentInvocations() {
        return currentInvocations;
    }
}

Now let’s define endpoint itself:

@Path("/time")
@Produces(MediaType.APPLICATION_JSON)
public class TimeResource {

    private final AtomicInteger invocations = new AtomicInteger(0);

    @GET
    public TimeAndInvocations getTimeAndInvocations() {
        int invocation = this.invocations.incrementAndGet();
        return new TimeAndInvocations(LocalTime.now(), invocation);
    }
}

This defines GET /time endpoint that returns JSON-formatted time and invocation count. All these annotations that allows to represent endpoint as java object come from JAX-RS (JSR 311 & JSR 339) and are implemented by Jersey.

However before we can hit our endpoint, we must let Dropwizard/Jersey know about it in HelloApplication class:

@Override
public void run(Configuration configuration, Environment environment) {
    environment.jersey().register(new TimeResource());
}

That’s it! Routing of incoming requests into Jersey environment is already done by Dropwizard and launching our app again confirms that:

The following paths were found for the configured resources:
  GET /time (com.romanmarkunas.dwtutorial1.TimeResource)```

Deserialization configuration

Let’s look what our endpoint returns:

{
    "time": [
        9,
        4,
        51,
        367000000
    ],
    "currentInvocations": 1
}

Jackson is the library that converts our TimeAndInvocations objects into JSON string. As you can see by default LocalTime serializer just prints fields.

Let’s fix this:

@JsonProperty
@JsonSerialize(using = LocalTimeSerializer.class)
@JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "hh:mm:ss")
public LocalTime getTime() {
    return time;
}

which results in output:

{
    "time": "09:45:15",
    "currentInvocations": 3
}

Here we added another Jackson annotation which hints serializer how it should form a string representation of that object. Jackson comes with plenty of serializers for Java library classes and other serializers can be registered into Jackson environment, similarly how we didi it with TimeResource. Otherwise, if Jackson doe not find specific serializer for and objects it works with, it will default to aforementioned field-by-field serialization.

Packaging

As mentioned before, Dropwizard promotes single JAR packaging. To make a single JAR we need to instruct build tool to put all depended upon classes into JAR file. I have covered this in separate tutorial before.

Instead of conclusion

That’s it for this introductory article. Hopefully after this reader will have vague ide of what Dropwizard is, how it may be used and roughly how it’s components are interacting to bring that.

Creating fat JARs with gradle

Tue, 13 Nov 2018 00:00:00 +0000

Here’s a quick tutorial on how to package your java application and dependencies into single fat JAR (shadow JAR, capsule, however it’s called) with gradle.

Easy solution using built-in JAR gradle plugin

jar {
    manifest {
        attributes(
            'Main-Class': "com.example.CraneBookingApplication"
        )
    }

   from {
        configurations.compile.collect { 
            it.isDirectory() ? it : zipTree(it) 
        } 
    }
}

What it means:

jar - packaging task available in gradle as part of java plugin
from - copies passed FileTree into JAR
configurations - holds all configurations (set of dependencies)
compile - I used compile configuration for this particular example, but this could be any other (i.g. implementation in newer Gradle versions) or even a separate configuration just for fat JAR
zipTree - this returns zipped file tree object, because JAR files essentially are ZIP files with specific file arrangement and content

While this is nice little solution that uses only built-in gradle stuff, it will work only in cases when there are no name clashes in all dependencies. For example, if more than one dependency implements Java SPI service interface you will have multiple META-INF resources with the same name and only last one will make it into fat JAR.

Using Shadow JAR plugin

buildscript {
    dependencies {
        classpath 'com.github.jengelman.gradle.plugins:shadow:4.0.1'
    }
}

apply plugin: 'com.github.johnrengelman.shadow'

shadowJar {
    mergeServiceFiles()
}

Internally this plugin does the same thing as previous one-liner, plus:

It collates SPI service files
inherits configuration from jar task so that you may define manifest in standard familiar way:

jar {
    manifest {
        attributes(
            'Main-Class': "com.example.CraneBookingApplication"
        )
    }
}

Using Shadow JAR plugin with Windows

There is important caveat for Windows users however with previous method. Shadow JAR uses \n as line separator when merging SPI files. Consequently, when JAR is run on Windows JVM will not be able to correctly read file line-by-line. To avoid this, we define custom service file transformer in build.gradle that will use \r\n as line separator:

import com.github.jengelman.gradle.plugins.shadow.relocation.Relocator
import com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer

class WindowsTransformer extends ServiceFileTransformer {

    @Override
    void transform(String path, InputStream is, List<Relocator> relocators) {
        def lines = is.readLines()
        
        relocators.each {rel ->
            if (rel.canRelocateClass(new File(path).name)) {
                path = rel.relocateClass(path)
            }
            lines.eachWithIndex { String line, int i ->
                if (rel.canRelocateClass(line)) {
                    lines[i] = rel.relocateClass(line)
                }
            }
        }
        
        lines.each { line -> serviceEntries[path]
                .append(new ByteArrayInputStream((line + "\r\n").getBytes()))
        }
    }
}

shadowJar {
//    mergeServiceFiles() - should not be used, as it uses default transformer
    transform(WindowsTransformer.class)
}

Kafka transactions in practice 1: Producer

Mon, 01 Oct 2018 00:00:00 +0100

This post will highlight some practical aspects of using kafka transactions. I will not dive deep into theory on how Kafka transactions are implemented. Instead, let’s discuss main “gotchas” someone could encounter when trying to use Kafka transactions in production first time.

For detailed theoretical reference please refer to this great article. Full code mentioned in article can be found here.

Transactional applications with Kafka

Transactions in Kafka can be divided into 3 big groups: transactions when producing messages, transactions when consuming and exactly once processing, when both consuming and producing are done within one single transaction.

In this post we will discuss only transactional producing.

Just one more note before we start. In this article we will discuss only transaction semantics, but not delivery semantics. These may differ, especially when transaction also involves systems external to Kafka.

Usage example

Let’s say we have an application that allows users to trade chairs between each other. Users submit buy/sell orders and our app finds matching orders so that trades can occur. We also want to aggregate order data to figure out daily volumes, most traded instruments and other fancy business intelligence data. However we do not want to submit customer specific data to business intelligence pipeline because it is personal information and should not be read by unauthorized personnel. One way to tackle that is to have a microservice that forwards orders to matcher queue and submits orders stripped of sensitive data into aggregator queue.

Successful scenario

To submit messages into different queues we use following code:

public void sendTransactionally(List<TopicAndMessage> messages) {
    producer.beginTransaction();
    
    for (TopicAndMessage topicAndMessage : messages) {
        producer.send(new ProducerRecord<>(
                topicAndMessage.getTopic(),
                getTransactionNr(),
                topicAndMessage.getMessage()));
    }

    producer.commitTransaction();
}

Now lets simulate successful write in our test scenario:

private static final List<TopicAndMessage> MESSAGES = asList(
            new TopicAndMessage(TRADE_MATCHER_TOPIC, "id: 123, buy 42 chairs"),
            new TopicAndMessage(AGGREGATOR_TOPIC, "buy 42 chairs"));

private KafkaConsumer<Integer, String> consumer = ... // create or get read_uncommitted consumer here
private KafkaProducer<Integer, String> producer = ... // create or get transactional producer here

@Test
public void noFailuresOccurred_readAllConsumer() {
    CrashableTransactionalKafkaProducer crashableProducer
            = new CrashableTransactionalKafkaProducer(producer);
    crashableProducer.sendTransactionally(MESSAGES);
    crashableProducer.sendTransactionally(MESSAGES);

    ConsumerRecords<Integer, String> records = poll();

    assertEquals(4, records.count());
    print(records);
}

Output:

Topic: orders, offset: 0, transaction: 1, message:  id: 123, buy 42 chairs
Topic: anonymous-orders, offset: 0, transaction: 1, message:  buy 42 chairs
Topic: orders, offset: 2, transaction: 2, message:  id: 123, buy 42 chairs
Topic: anonymous-orders, offset: 2, transaction: 2, message:  buy 42 chairs

Looks good! Very attentive reader will notice that 2nd transaction offsets start from 2 even though we submitted only 1 message before that. Also what does read_uncommitted mean in comment above? Let’s discuss this in a moment after checking out failure scenario.

Transactional KafkaProducer initialisation is stuck

If something is set incorrectly, most of the time Kafka will throw descriptive exception, for example:

java.lang.IllegalStateException: Cannot use transactional methods without enabling transactions by setting the transactional.id configuration property

However one setting may cause indefinite blocking and “hanging”. This is broker setting transaction.state.log.min.isr (see also transaction.state.log.replication.factor when changing this). To keep track of which transactions were committed Kafka keeps internal topic called __transaction_state. This topic should be replicated for fault-tolerance as any other with Kafka. Now when you call initTransactions() on producer, this call will block until topic has all in-sync replicas acknowledgement.

For example, if broker has only 2 machines online and transaction.state.log.min.isr = 3, initTransactions() will block until one more broker node gets online. If that 3rd node will go offline after producer has been initialized successfully, everything will keep working. However for startup it is necessary to have minimum broker node count online.

Failure scenario

Now following scenario:

@Test
public void crashOnSecondTransaction_readAllConsumer() {
    crashableProducer.sendTransactionally(MESSAGES);
    crashableProducer.setShouldCrash(true);
    try {
        crashableProducer.sendTransactionally(MESSAGES);
    }
    catch (ForceCrashException e) {
        System.out.println("Producer crashed!");
    }

    List<ConsumerRecord<Integer, String>> records = poll();

    assertEquals(3, records.size());
    print(records);
}

gives us:

Producer crashed!
Topic: orders, offset: 0, transaction: 1, message:  id: 123, buy 42 chairs
Topic: anonymous-orders, offset: 0, transaction: 1, message:  buy 42 chairs
Topic: orders, offset: 2, transaction: 2, message:  id: 123, buy 42 chairs

Oops! Looks like transactions didn’t work. But they actually do. The problem here is that consumer was created with isolation.level = read_uncommitted setting, which is it’s default value. Every time producer commits a transaction it submits a commit marker to the log. If consumer’s isolation.level is set to read_uncommitted it just reads all messages it can. In case setting is read_committed consumer will wait until commit marker is submitted to decide what to do with preceding messages.

This commit marker contains all necessary data for consumer to read from __transactions_state topic and decide, if message must be returned. In case when producer abort transaction commit marker will represents abort and will be ignore by consumer. In case producer dies, broker will ultimately expire transaction and abort marker itself. This is why if you submit n messages to topic the next batch will always start at n+1 offset.

Now if we set consumer’s isolation.level = read_committed, we will get:

Producer crashed!
Topic: orders, offset: 4, transaction: 1, message:  id: 123, buy 42 chairs
Topic: anonymous-orders, offset: 2, transaction: 1, message:  buy 42 chairs

So important note here is that transactional producing is dependent on the readers. Therefore when building pipelines with Kafka, every node in chain settings must be carefully considered, because one error may cause change in all pipeline delivery semantics!

Producer fencing and expiration

To protect cluster from intermittent connection errors Kafka uses epochs tied to producer’s transaction ID. Every time producer calls initTransactions(), cluster will assign it an epoch = previous_epoch + 1. This means that if you start producer with transaction id and cluster has active producer with same id, the latter one will be disconnected (fenced in Kafka terms) and throw an exception next time it tries to send message:

org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an 
operation with an old epoch. Either there is a newer producer with the same 
transactionalId, or the producer's transaction has been expired by the broker.

Sometimes producer throws another exception, with ProducerFencedException as cause.

But this exception may occur even without connectivity errors. Internally broker keeps a map of producer IDs to transaction IDs to maintain aforementioned fencing functionality. However broker will expire these entries based on transactional.id.expiration.ms setting which is by default 604800000 ms or 1 week.

This means that if your producer sends messages slower than once a week it will be fenced (crash) every time it tries to send after that prolonged period. One could put that value to Integer.MAX, but that still will be around 24 days. In case of very rare events, this should be solved differently, e.g. by having separate producer ping topic, that producer will periodically send messages to.

Instead of conclusion

That’s it for Kafka producers! Hopefully this will help you to understand, setup Kafka transactions and successfully keep them producing. For more implementation details how Kafka transactions work, I recommend blog mentioned in this post’s intro and this Kafka KIP.

Stay tuned for article on transactional consuming!

Notes on Java Random

Mon, 01 Oct 2018 00:00:00 +0100

This article is inspired by Monty-Hall problem/paradox. Now I found this paradox disturbing initially (this is why it’s paradox, wink…), and decided to quickly model it to validate for myself. That brought idea of writing up all I know about Java Random API in this article.

If you want to read about paradox itself click here

java.util.Random

Instantiation

Random generator is a bit group sequence generator. Random constructor accepts “seed” parameter which is just a starting value for the sequence generator. This is why if you seed 2 Random instances with same number and call same next() methods in exact sequence, generated number will be exactly the same.

To protect default Random constructor from such behavior, Java uses current system time to seed generator. However it is possible that 2 threads will instantiate Random objects at exactly same nano second, therefore current time is XOR’ed with “seed uniquifier” value which is guaranteed to be unique for each thread via AtomicLong#compareAndSet().

Please note, that if you need multiple threads served by random generator it might be beneficial to use ThreadLocalRandom, that reduces thread contention.

Number generation

Actual “random” number generation is done by next() method as all other methods are just wrappers around that to create correct variable, using some bit arithmetic.

The next() method itself is multiplication, addition and masking with carefully selected values that ensure good uniformity. This means that once you have same bits generated by 2 random generators (which could be achieved via setSeed() method), next generated sequences will be exactly the same, similar to what is achieved by having same seed from the start.

Number predictability

At this point you may feel that aforementioned predictability may break security of system, if Random is used to generate tokens, passwords, IDs, etc. In fact you can calculate Random seed using just 2 subsequent generated values. Also since next() returns only 32 bytes, having longer (e.g. 128-bit) token will not improve security.

When to use

Use Random or ThreadLocalRandom when you care only about uniformity of generated numbers. These will give you uniform probabilities and decent performance.

java.security.SecureRandom

As said before biggest problem with Random is that it’s reversible. To break this we may use cryptographically strong (requires many iterations to guess input based on output) hashing function and apply it to generated numbers. In fact SecureRandom is a mini-framework to assemble you own generator from seeding, byte generation and hashing function.

Structure

SecureRandom is not a generator itself, but a Template pattern around how random generator must operate. It uses Java SPI functionality, that allows you to hook in any implementation of SecureRandomSpi that would be used to actually generate seed and bytes. Other than that, SecureRandom operates similarly to Random.

In case you need to use different generators in the same application, you can use constructors/factory methods with your implementation or Provider to get SecureRandom instances with different backends.

So what if you don’t provide your own SPI implementation? Then SecureRandom will fall back to default Sun implementation - sun.security.provider.SecureRandom. Let’s further describe what this Sun implementation does to generate random numbers.

Initialization

To obtain random seed Sun implementation uses SeedGenerator object. Here you have 3 options:

You can specify a system property to read seed from your URL. In this case URLSeedGenerator instance will be used
If you don’t go 1st option it will attempt to use NativeSeedGenerator (more on this below)
If your OS does not support NativeSeedGenerator, it will fall back to using ThreadSeedGenerator, which will generate seed based on how much time OS takes to switch between application threads.

NativeSeedGenerator is the instance that most likely to be used. This generator will use random device of your OS/hardware:

/dev/random/ or /dev/urandom on Unix-likes
Crypto API on Windows These devices generate randomness based on user/OS events, e.g. interrupts (IO events such as keystrokes and network events, hardware timers, etc). The exact device used may be found in $JAVA_HOME/lib/security/java.security file under securerandom.source property.

Why would URLSeedGenerator be necessary? It could be used to implement true random number generator, e.g. you use weather forecast to generate randomness. This data may be supplied as web service and your random generator may use URL of that service to seed itself.

Number generation

Number generation is similar to Random, however every generated number gets hashed using algorithm that is specified on SecureRandom creation. This way if attacker gets hold of generated values, it is harder to figure out number generation algorithm as they also have to brute force values to get original ones.

When to use

Use when you need to generate anything that is security-related: temporary passwords, session IDs, tokens, etc.

/dev/random blocking

I though I write this up separately, as finding this took N man-hours to find out.

If your system uses /dev/random, it will block until minimal level of randomness is generated. Normally OS save some randomness before restart and no blocking happens. However, if you launch containerized application, there is nothing stored since last launch and you will have to wait. This may take very long time (I saw this taking over 1 hour on Amazon EC instance), especially if you run it on server setting when no one is pressing keyboard.

Be wary that some Java components use SecureRandom under the hood, e.g. UUID (when it generates IDs) or File (when creating temporary files), so you may encounter this in situations outside security domain.

In place of conclusion

Hopefully this sheds some light on how Java random number generators work under the hood. Thanks for reading and stay safe!

Monty-Hall problem

Mon, 01 Oct 2018 00:00:00 +0100

This is totally random write-up and it made it here because it inspired post about Java Random number generation. Here is how it goes…

There are 3 doors and behind each is either a car or a goat. There are 2 goats and 1 car altogether. If you try to guess behind which door is a car, what is the chance of getting it right? Easy! 1 out of 3 makes 33,3% chance of getting it right.

If someone would open a door with a goat (but not the one that you initially chose) and ask you if you want to change your selection? Would you change your choice or stay with initial door? The correct answer is… if you change door you will have 66,6% of getting a car, but if you stay with initial choice - you stay with initial 33,3% chance.

Now I though this result is awkward, because when one door opened you have 2 doors to choose, which is essentially 50/50 chance. But the model (which can be found here proves this is incorrect. So Monty-Hall problem suggests that probabilities have temporal linking as well, meaning probability depends on the moment when decision is made as well as your odds.

Messaging system latencies, part 1: Apache Kafka

Sat, 22 Sep 2018 00:00:00 +0100

Most developers I talk to about Kafka agree on a catchphrase “Kafka is designed for throughput”. That’s fair and you can find plenty of benchmarks that show 700k/s throughput for single producer without replication. But does that mean we should discard Kafka when talking about low latency messaging?

After quick googling I found this outdated SO question. That question is very old and includes very old version of Kafka (0.7.2 whereas current version is 2.0.0) and also this article states that very decent latencies (2-3ms 99 percentile) are achievable. Having such controversial info is not enough to make final decision so I decided to create a little benchmark myself, to finally conclude whether Kafka is good for low-latency applications.

What is measured

For all tests I run producer, consumer and broker on the same machine. For these tests I used laptop: i7-7820HQ, 16GB, Windows 10 (yes, I know…). This is why pre-allocation is on in broker settings and you will need to disable it should you run provided code on Linux box. Also I didn’t change default Kafka startup scripts, which means brokers had 1GB heap maximum.

Latency test intent is to test following scenarios:

light throughput of 200 messages/second + non-durable broker
light throughput of 200 messages/second + fault-tolerant broker
moderate throughput of 4000 messages/second + non-durable broker
moderate throughput of 4000 messages/second + fault tolerant broker

This test do not measure latency drops due to cluster node failovers as these scenarios are very different depending on your partitioning schema and replication factor. Hopefully node failures are not part of your normal day-to-day operations ;)

As usual, all code can be found here, if you want to play around and see how your setup compares. All scenarios are located in test directory under benchmark/LatencyBenchmark.

For exact broker/client configuration see code above. Also I’ll put a little explanation why these settings were used.

Results

Lowest latencies possible

Measurements @ 200 messages/s:

Total sent     - 5000
Total received - 5000
Send rate      - 200.004
99 percentile  - 1.576891
75 percentile  - 1.180825
Min latency    - 0.593806
Max latency    - 8.921906
Avg latency    - 1.085105

Measurements @ 4000 messages/s:

Total sent     - 50000
Total received - 50000
Send rate      - 3837.725
99 percentile  - 1.327242
75 percentile  - 0.945013
Min latency    - 0.413972
Max latency    - 13.287300
Avg latency    - 0.802173

So these are lowest latencies possible on my machine. Configuration for these:

No batching on producer (consumer always fetches all available messages up to configured size)
No acks on producer
No commits after each message (but uses asynchronous background commits) on consumer
Single node cluster, no replication
Broker has delayed sync to disk
All internal topics are single partition

As you can see most messages make a roundtrip within 1-2 ms. There are always outliers at rare occasion, which are caused by different maintenance operations and probably some interference from OS. For example, expired offset removal took around 7 ms.

Also it’s possible to decrease consumer poll timeout to get smaller max latency at cost of higher 99 percentile.

Non fault-tolerant setup with consumer commit after each read

Measurements @ 200 messages/s and synchronous commit:

Total sent     - 5000
Total received - 5000
Send rate      - 200.004
99 percentile  - 4.535680
75 percentile  - 1.927668
Min latency    - 0.904814
Max latency    - 9.848230
Avg latency    - 1.865007

I made these measurements as I was curious how synchronous commit affects latency, and it’s roughly 30% overhead.

Fault-tolerant setup with synchronous consumer commit

Measurements @ 200 messages/s and synchronous commit:

Total sent     - 5000
Total received - 5000
Send rate      - 199.989
99 percentile  - 5.137226
75 percentile  - 3.924183
Min latency    - 2.115347
Max latency    - 14.580346
Avg latency    - 3.613249

Measurements @ 4000 messages/s and synchronous commit:

Total sent     - 50000
Total received - 50000
Send rate      - 3993.356
99 percentile  - 17.898772
75 percentile  - 11.823148
Min latency    - 5.039950
Max latency    - 35.926347
Avg latency    - 11.018564

Fault tolerant config:

Broker made out of 3 nodes
Producer waits for ack from each broker node
Sync to disk is still delayed

For the higher throughput scenario I had to turn batching on and put a setting to batch at most 500 records. Otherwise maximum throughput would be 1000 / MIN_LATENCY = 1000 / 5 = 200 messages/s

Conclusion

Looks like these articles about huge latencies of 100ms magnitude relate to older versions of Kafka. Replicated operation showed pretty decent results, which should be enough for most cases. Please note that these results are not made on dedicated hardware and I didn’t do any memory/OS settings/affinity tuning, so very likely it’s possible to get even better results.

There are no measurements for durable messages (force flush each message on disk) and I think tests for such setup are extremely hardware dependent and should be made using proper HDD/SSD.

My takeaway from this is that unless you need sub-millisecond latencies I could do with Kafka. Unless you are in that outlier part of spectrum technologies/solution selection must be done based on functionality it provides and not on specific performance parameters. And of course reusing well-known technology is much better than having a zoo of trendy names, that barely anyone in team understands properly.

P.S.

Coming soon! Follow-up post where same tests are performed with Apache Artemis (former ActiveMQ).