After upgrading to brXM v15 the repository directory is created in the wrong location

When attempting to upgrade to v15 of the Bloomreach Experience Manager we ran into two observable issues.

1. The repository directory is create one directory up from where it used to. So for example if it had been originally created in /dir-1/dir-2/dir-3/project/repository, it's now created in /dir-1/dir-2/dir-3/repository instead.
2. It appears that webfiles are not being bootstraped properly even though that there does not seem to be any indication from the logs of a specific error or problem with the CMS loading or with the bootstrapping process.

This issue has only occured in the four production/staging nodes that we've attempted so far but locally and in our several other dev and test servers we've been unable to reproduce the problem. I've detailed the overall issue in the community forum too which I've linked below.

Webfile changes fail to bootstrap for upgrade from v14 to v15

For the purpose of this issue though I'd like to focus on #1 above since it is easily observable, hopefully easier to troubleshoot and could potentially be related to the issue described as #2 above.

Details

Originally in context.xml we've had the repository-directory defined as shown below.

<Parameter name="repository-directory" value="${catalina.base}/../repository" override="false"/> 

This appears to align with the guidance given in the Linux Installation Manual and hadn't been modified in our production environments since Dec 29 2021. However and as mentioned before, after deploying the application with the v15 upgrade, the repository directory was created one directory up from where it was expected.

Expected directory:

/vendor/APACHE/TOMCAT/tomcat_base/repository

Actual directory:

/vendor/APACHE/TOMCAT/repository

Potential causes

Since the repository-directory property as specified in the context.xml file hadn't changed, my conclusion was that either this property hadn't been used prior to v15 even though it was specified or that the effective value of catalina.base had changed.

Verifying catalina.base

To get more clarity on whether the value for catalina.base had changed in some way within our deployment, with the new Java 11 environment, or in the initialization handling within brXM v15 I took a look at the value through the systemproperties view in the CMS for both an active running version of v14 in production and the node with the issue running v15 and verified that both resolved to the value of /vendor/APACHE/TOMCAT/tomcat_base. This confirms that according to the value of , that the value of repository-directory should have resolved to /vendor/APACHE/TOMCAT/tomcat_base/../repository or just /vendor/APACHE/TOMCAT/repository both before and after the upgrade leaving only the theory that this repository-directory value wasn't being used prior to the upgrade or was somehow being overridden.

Attempted workaround with context.xml

We tried to remedy the issue by changing the repository-directory parameter directly in context.xml to the value below.

<Parameter name="repository-directory" value="${catalina.base}/repository" override="false"/> 

Coincidentally, this happened to match the configuration in our lower environments which have seemed to have been immune to the bootstrapping and repository directory resolution issue at hand. Additionally this did seem to correct the issue in the v15 upgrade so that the repository directory is created in the same place it has prior for v14, it didn't seem to help us at all in terms of resolving issue #2 with regard to the webfiles bootstrapping problem. At this point as well we are left unsure of why repository-directory resolution is working differently before and after the upgrade.

Summary

We are left at this point to conclude that something in our production environment specific to the v15 upgrade which could include behavior or configuration differences between Java 8 and Java 11 on the machine (ex. JAVA_OPTS, ...) or something related to the behavior of how the repository-directory property is used from context.xml and ultimately how the effective `repo.dir` value is derived has changed between v14 to v15 in a way that seems to be undocumented and which seems to be having an impact on the startup and potentially the bootstrapping processes during the upgrade.

We have log files that have been captured recently which isolates the startup and upgrade attempt but since the log files contain production environment related information and are also in excess of 10MB (~60MB total compressed), we'll need some other way to share these logs to anyone who would be able to assist us in troubleshooting the issue.{}