The observed behavior is really subtle and hard to reproduce. The following setup is needed:

1. a channel with many hst jcr nodes, say 1.000
2. run the application with a BRC garbage collector configuration like -XX:G1HeapRegionSize=4m -XX:+UseG1GC
3. have a bundle cache size which should be able to cache quite some JCR Nodes

The following happens when now branch the large channel (resulting a branch containing ~3.000 jcr nodes as there is live, upstream and preview)

1. The JCR Nodes are written in org.apache.jackrabbit.core.persistence.bundle.AbstractBundlePersistenceManager#putBundle but only the JCR Nodes which already existed before in the bundle cache are updated, see

           // only put to cache if already exists. this is to ensure proper
           // overwrite and not creating big contention during bulk loads
           if (bundles.containsKey(bundle.getId())) {
               bundles.put(bundle.getId(), bundle, bundle.getSize());
           }
   

2. As the new JCR Nodes are not in bundle cache, in general they are for some time in the ItemStateReferenceCache from the SharedItemStateManager, however, this cache is a cache with fixed max size of 4 * 1024 * 1024 (4 Mb which hardly fits many jcr nodes) but also, this is a weak reference cache

It turns out because of the above, that the just persisted JCR Nodes are NOT in the bundle cache and not in the SharedItemStateManager when the HST model has to be loaded. Then the HST model loading, which is blocking many operations, takes considerable time to fetch all the just persisted JCR Nodes from the database.

Solution

Apart from that I find it typical that the SharedItemStateManager has a cache of only 4Mb, (an equally sized cache as the LocalItemStateManager which surprises me), it seems best to me to cache the newly created jcr nodes in the bundle cache as it is very likely that right after creating the jcr nodes, the very same jcr nodes need to be read. At this moment, that only works when the amount of jcr nodes is not too large or when read before they are garbage collected from the SharedItemStateManager cache

To remedy this, we can better remove the code :

if (bundles.containsKey(bundle.getId())) {
            bundles.put(bundle.getId(), bundle, bundle.getSize());
        }

altogether and instead do this in one go for all modified/added states. The reason I prefer to do it in one go is that every separate bundle put can trigger some cache overhead to shrink it first before the new entry can be added: we do not want the cache to have to shrink, say, 3000 times, if 3000 new nodes are added. Instead we can temporarily set the cache size lower, which triggers automatic shrinking, and then increase the cache size again and then add all the new jcr nodes to the bundle cache. Something like this:

// only for an update of larger than 8 Mb, we first shrink the cache actively
        if (updateSize > 8 * 1024 * 1024) {
            final long maxMemorySize = bundles.getMaxMemorySize();
            // never shrink below 8 * 1024 * 1024
            final long temporaryMaxMemorySize = Math.max(8 * 1024 * 1024, maxMemorySize - updateSize);
            log.info("Shrinking the bundle cache with {} to {}", updateSize, temporaryMaxMemorySize);
            // this triggers the shrinking of the cache in one go such that it can accomodate the newly added
            // node states without having to shrink very frequently during adding the newly added nodes
            log.info("Elements in cache before shrinking : {}", bundles.getElementCount());
            bundles.setMaxMemorySize(temporaryMaxMemorySize);
            log.info("Elements in cache after shrinking : {}", bundles.getElementCount());
            // reset to original max size: now we create space for the added / modified states to be added
            bundles.setMaxMemorySize(maxMemorySize);
        }
        // cache the newly added / changed nodes
        for (NodePropBundle bundle : modified.values()) {
            bundles.put(bundle.getId(), bundle, bundle.getSize());
        }