Repository consistency check does not fix all inconsistencies

This is very disruptive as it requires a rollback of the upgrade which involves restoring the database from backup, restarting the (old) version and manually track down the broken nodes. This process usually takes hours of manual work.

Running an upgrade with:
<param name="consistencyCheck" value="true"/>
<param name="consistencyFix" value="true"/>

26.04.2011 21:44:14 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkConsistency():235] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 64000/71742 bundles...
26.04.2011 21:44:17 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkConsistency():235] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 65000/71742 bundles...
26.04.2011 21:44:19 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkConsistency():235] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 66000/71742 bundles...
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016419107' with id 'ab8515c8-c0ce-41
b9-9c7a-6a61932b11d0'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016419326' with id 'dbd2ece7-f37f-408b-98af-d2cad28b82f9'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016464282' with id '7ac47613-9c8b-4389-98e6-bf5252009f68'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016467309' with id '9f63cbd7-edb7-4fe1-abd9-8a5561673157'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016471180' with id '7528978f-c46f-482d-b034-3eb1db9dc44b'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016471395' with id '1e065a05-71f4-4370-87c5-35478d451188'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016510329' with id '2cdfa2d4-cd32-4042-9b58-fb49cfd56bc1'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016522330' with id '85a7960a-525a-48bd-b591-cbd7d46921d2'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016525030' with id 'a47c5343-4ad4-4df8-abea-e4718d5b0260'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016535090' with id 'ecf15d15-beb1-48ea-a250-bbd97d54886b'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016542312' with id 'd826ff98-996e-49d7-950d-46a1d54c9a3b'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016544169' with id 'a6f5348c-3c63-4f1d-911d-cbe1360c4df7'
26.04.2011 21:44:20 ERROR [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkBundleConsistency():110] NodeState 'ec0d0368-4803-4f19-b7d1-51963b558dab' references inexistent child '{}1302016546300' with id '4e8b9a78-5387-44a9-9311-74ae2976cd88'
26.04.2011 21:44:22 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkConsistency():235] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 67000/71742 bundles...
26.04.2011 21:44:25 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkConsistency():235] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 68000/71742 bundles...
26.04.2011 21:44:27 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkConsistency():235] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 69000/71742 bundles...
26.04.2011 21:44:29 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkConsistency():235] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 70000/71742 bundles...
26.04.2011 21:44:32 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkConsistency():235] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 71000/71742 bundles...
26.04.2011 21:44:33 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkConsistency():297] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: Fixing 1 inconsistent bundle(s)...
26.04.2011 21:44:33 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkConsistency():300] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: Fixing bundle 'ec0d0368-4803-4f19-b7d1-51963b558dab'
26.04.2011 21:44:34 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkConsistency():324] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 71742/71742 bundles.
26.04.2011 21:44:34 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkReferences():341] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checking references consistency...
26.04.2011 21:44:34 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkReferences():359] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 0/2092 references...
26.04.2011 21:44:37 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkReferences():384] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 1000/2092 references...
26.04.2011 21:44:40 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkReferences():384] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 2000/2092 references...
26.04.2011 21:44:41 INFO [org.hippoecm.repository.jackrabbit.persistence.PatchedBundleDbPersistenceManager.checkReferences():393] org.hippoecm.repository.jackrabbit.persistence.ForkedMySqlPersistenceManager@67e779a1: checked 2092/2092 references.

Still results in itemnotfoundexceptions:
26.04.2011 21:44:45 INFO [org.hippoecm.repository.updater.UpdaterEngine.upgrade():678] upgrade cycle (phase 2 of 5) traverse process
26.04.2011 21:44:45 INFO [org.hippoecm.repository.updater.UpdaterEngine.process():945] migration update traverse for module CustomerUpdateModule
26.04.2011 21:44:45 INFO [org.hippoecm.repository.updater.UpdaterEngine.upgrade():685] upgrade cycle (phase 3 of 5) iterated process
26.04.2011 21:44:45 INFO [org.hippoecm.repository.updater.UpdaterEngine.upgrade():689] migration update cycle for module CustomerUpdateModule
26.04.2011 21:44:45 INFO [org.hippoecm.repository.updater.UpdaterEngine.upgrade():704] migration update iterated for module CustomerUpdateModule (PathVisitor[/hippo:namespaces]) 1 nodes
26.04.2011 21:44:45 INFO [org.hippoecm.repository.updater.UpdaterEngine.upgrade():704] migration update iterated for module CustomerUpdateModule (PathVisitor[/hippo:configuration/hippo:initialize]) 1 nodes
26.04.2011 21:44:45 INFO [org.hippoecm.repository.updater.UpdaterEngine.upgrade():704] migration update iterated for module CustomerUpdateModule (PathVisitor[/hst:configuration/hst:configuration]) 0 nodes
26.04.2011 21:44:45 INFO [org.hippoecm.repository.updater.UpdaterEngine.upgrade():704] migration update iterated for module CustomerUpdateModule (PathVisitor[/content/documents]) 1 nodes
26.04.2011 21:44:47 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: ee29c31d-c068-4e22-b747-c56f56d747cc: javax.jcr.ItemNotFoundException: ee29c31d-c068-4e22-b747-c56f56d747cc
26.04.2011 21:44:47 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: e4e510c5-47c3-47a4-9d03-85bd3c7bd89c: javax.jcr.ItemNotFoundException: e4e510c5-47c3-47a4-9d03-85bd3c7bd89c
26.04.2011 21:44:47 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: b727369e-a3e2-477f-8aed-2be7d104bdb8: javax.jcr.ItemNotFoundException: b727369e-a3e2-477f-8aed-2be7d104bdb8
26.04.2011 21:44:47 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: 237074ed-d8be-4bfb-a27e-edf4002f666c: javax.jcr.ItemNotFoundException: 237074ed-d8be-4bfb-a27e-edf4002f666c
26.04.2011 21:44:47 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: 223ebba0-c480-49fc-aafe-59b8cd5a00fa: javax.jcr.ItemNotFoundException: 223ebba0-c480-49fc-aafe-59b8cd5a00fa
26.04.2011 21:44:48 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: ff5aab37-5635-453c-a4cf-df45cd31fa15: javax.jcr.ItemNotFoundException: ff5aab37-5635-453c-a4cf-df45cd31fa15
26.04.2011 21:44:48 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: fc7ac8ea-116a-4c56-96ca-08910e70c021: javax.jcr.ItemNotFoundException: fc7ac8ea-116a-4c56-96ca-08910e70c021
26.04.2011 21:44:50 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: e1f56146-23e5-491b-b163-723cb093f107: javax.jcr.ItemNotFoundException: e1f56146-23e5-491b-b163-723cb093f107
26.04.2011 21:44:55 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: ee29c31d-c068-4e22-b747-c56f56d747cc: javax.jcr.ItemNotFoundException: ee29c31d-c068-4e22-b747-c56f56d747cc
26.04.2011 21:44:55 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: e4e510c5-47c3-47a4-9d03-85bd3c7bd89c: javax.jcr.ItemNotFoundException: e4e510c5-47c3-47a4-9d03-85bd3c7bd89c
26.04.2011 21:44:55 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: b727369e-a3e2-477f-8aed-2be7d104bdb8: javax.jcr.ItemNotFoundException: b727369e-a3e2-477f-8aed-2be7d104bdb8
26.04.2011 21:44:55 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: 237074ed-d8be-4bfb-a27e-edf4002f666c: javax.jcr.ItemNotFoundException: 237074ed-d8be-4bfb-a27e-edf4002f666c
26.04.2011 21:44:55 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: 223ebba0-c480-49fc-aafe-59b8cd5a00fa: javax.jcr.ItemNotFoundException: 223ebba0-c480-49fc-aafe-59b8cd5a00fa
26.04.2011 21:44:56 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: f487d9ce-973b-4a8a-9391-55a0a9c833bd: javax.jcr.ItemNotFoundException: f487d9ce-973b-4a8a-9391-55a0a9c833bd
26.04.2011 21:44:56 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: c93f5aa4-a219-4c34-9a64-84f94db8c113: javax.jcr.ItemNotFoundException: c93f5aa4-a219-4c34-9a64-84f94db8c113
26.04.2011 21:44:56 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: e4e510c5-47c3-47a4-9d03-85bd3c7bd89c: javax.jcr.ItemNotFoundException: e4e510c5-47c3-47a4-9d03-85bd3c7bd89c
26.04.2011 21:44:56 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: b727369e-a3e2-477f-8aed-2be7d104bdb8: javax.jcr.ItemNotFoundException: b727369e-a3e2-477f-8aed-2be7d104bdb8
26.04.2011 21:44:56 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: 223ebba0-c480-49fc-aafe-59b8cd5a00fa: javax.jcr.ItemNotFoundException: 223ebba0-c480-49fc-aafe-59b8cd5a00fa
26.04.2011 21:44:56 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: b727369e-a3e2-477f-8aed-2be7d104bdb8: javax.jcr.ItemNotFoundException: b727369e-a3e2-477f-8aed-2be7d104bdb8
26.04.2011 21:44:56 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: 223ebba0-c480-49fc-aafe-59b8cd5a00fa: javax.jcr.ItemNotFoundException: 223ebba0-c480-49fc-aafe-59b8cd5a00fa
26.04.2011 21:44:56 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: ee29c31d-c068-4e22-b747-c56f56d747cc: javax.jcr.ItemNotFoundException: ee29c31d-c068-4e22-b747-c56f56d747cc
26.04.2011 21:44:56 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: 237074ed-d8be-4bfb-a27e-edf4002f666c: javax.jcr.ItemNotFoundException: 237074ed-d8be-4bfb-a27e-edf4002f666c
26.04.2011 21:44:56 WARN [org.apache.jackrabbit.core.query.lucene.NodeIteratorImpl.fetchNext():185] Exception retrieving Node with UUID: e4e510c5-47c3-47a4-9d03-85bd3c7bd89c: javax.jcr.ItemNotFoundException: e4e510c5-47c3-47a4-9d03-85bd3c7bd89c
26.04.2011 21:44:56 INFO [org.hippoecm.repository.updater.UpdaterEngine.upgrade():704] migration update iterated for module CustomerUpdateModule (NamespaceVisitor[dcr]) 12973 nodes
26.04.2011 21:44:57 ERROR [org.hippoecm.repository.updater.UpdaterEngine.migrate():652] error in migration cycle: javax.jcr.ItemNotFoundException: failed to build path of be1d41f4-3277-43f4-a8a2-a21fe5948f44: fb440f4f-60de-430d-95b7-c195f303e951 has no child entry for be1d41f4-3277-43f4-a8a2-a21fe5948f44
javax.jcr.ItemNotFoundException: failed to build path of be1d41f4-3277-43f4-a8a2-a21fe5948f44: fb440f4f-60de-430d-95b7-c195f303e951 has no child entry for be1d41f4-3277-43f4-a8a2-a21fe5948f44
at org.apache.jackrabbit.core.HierarchyManagerImpl.buildPath(HierarchyManagerImpl.java:295)
at org.apache.jackrabbit.core.CachingHierarchyManager.buildPath(CachingHierarchyManager.java:198)
at org.apache.jackrabbit.core.HierarchyManagerImpl.getPath(HierarchyManagerImpl.java:399)
at org.apache.jackrabbit.core.CachingHierarchyManager.getPath(CachingHierarchyManager.java:232)
at org.hippoecm.repository.jackrabbit.HippoHierarchyManager.getPath(HippoHierarchyManager.java:64)
at org.apache.jackrabbit.core.ItemImpl.getPrimaryPath(ItemImpl.java:216)
at org.apache.jackrabbit.core.NodeImpl.getPrimaryPath(NodeImpl.java:3072)
at org.apache.jackrabbit.core.ItemImpl.getPath(ItemImpl.java:1315)
at org.hippoecm.repository.decorating.ItemDecorator.getPath(ItemDecorator.java:91)
at org.hippoecm.repository.updater.UpdaterEngine.upgrade(UpdaterEngine.java:707)
at org.hippoecm.repository.updater.UpdaterEngine.migrate(UpdaterEngine.java:620)
at org.hippoecm.repository.LocalHippoRepository.initializeStartup(LocalHippoRepository.java:378)
at org.hippoecm.repository.LocalHippoRepository.initialize(LocalHippoRepository.java:278)
at org.hippoecm.repository.LocalHippoRepository.<init>(LocalHippoRepository.java:155)