instance upgrade does not apply startup resources from the new Quickstart JAR

[royteeuwen]

Summary

aem instance upgrade unpacks a new Quickstart distribution, but the subsequent normal aem instance start launches the unpacked JAR under crx-quickstart/app.

For AEM 6.5 LTS service-pack upgrades, this does not apply all startup-resource updates from the new Quickstart distribution.

Reproduction

1. Create and start an AEM 6.6.2 instance through AEMC.
2. Replace the configured Quickstart JAR with AEM 6.6.3.
3. Stop the instance.
4. Run:

aem instance upgrade
aem instance start

Actual Result

AEMC unpacks the 6.6.3 distribution, then the normal start script launches:

crx-quickstart/app/cq-quickstart-6.6.3-standalone-quickstart.jar

The prior startup resource remains active:

crx-quickstart/launchpad/startup/1/com.adobe.granite.crypto-3.5.10.jar

AEM 6.6.3 upgrades com.adobe.granite.crypto.keystore to 4.0.24, but it remains Installed because it requires the package exported by Granite Crypto 3.5.40.

This causes a broad OSGi dependency cascade, including com.day.cq.cq-replication.

Expected Result

During an upgrade, AEMC should start the external new Quickstart JAR once, instead of only starting the unpacked app JAR:

java -jar <new-quickstart-jar> -nofork -nointeractive \
  -r <run-modes> -p <port>

After this bootstrap start has completed, AEMC can stop it. The caller can then use the ordinary aem instance start lifecycle.

Verified Result

With that external-JAR bootstrap:

com.adobe.granite.crypto           3.5.40  Active
com.adobe.granite.crypto.keystore  4.0.24  Active
com.day.cq.cq-replication          6.2.86  Active
org.apache.commons.commons-io      2.21.0  Active

All bundles become stable.

Adobe Reference

Adobe explicitly states that starting through the normal start script does not start the upgrade:

https://experienceleague.adobe.com/en/docs/experience-manager-65-lts/content/implementing/deploying/upgrading/in-place-upgrade#determining-the-correct-upgrade-start-command:~:text=start%20script%20will%20not%20start%20the%20upgrade

Sling Properties

Adobe's preparation instructions also say to back up and delete:

crx-quickstart/conf/sling.properties

before unpacking the new Quickstart JAR.

AEMC currently manages custom Sling properties itself. We need a design decision on whether instance upgrade should:

1. Back up/remove and regenerate sling.properties as part of the upgrade command.
2. Leave this preparation step to the caller/orchestration layer before invoking instance upgrade.

[royteeuwen]

Opened PR #356 for the external Quickstart bootstrap flow: #356

@krystian-panek-vmltech , could you please review the proposed upgrade lifecycle?

The handling of crx-quickstart/conf/sling.properties remains an open design question. Adobe recommends backing it up and deleting it before an in-place upgrade, but AEMC manages selected properties through aem.yml. Users may also have manually added properties that AEMC cannot recreate, so we should decide whether the upgrade command should reset/regenerate it or require caller/orchestration handling.

[krystian-panek-vmltech]

During an upgrade, AEMC should start the external new Quickstart JAR once, instead of only starting the unpacked app JAR:

I guess it is not true, AEMC always uses external quickstart JAR - is never directly using unpacked app jar

sling.properties handling seemed to be covered already, I remember that I was fixing this, just removing it when doing upgrade was enough (or sth like that). my teammmates already have done few upgrades with that command successfully, so ... I am not sure if your issues are valid, tbh ;)

[krystian-panek-vmltech]

I have to double check your issues. I will get back to you with this next week.

[royteeuwen]

Thanks, I may have described the distinction imprecisely.

AEMC does use the external distribution Quickstart JAR for -unpack. However, after that, the normal instance start path calls crx-quickstart/bin/start. That script resolves app/*.jar and runs:

java ... -jar crx-quickstart/app/<quickstart-jar>

It does not invoke the external distribution JAR.

Our 6.6.2 -> 6.6.3 test only completed cleanly when we launched the new external JAR once after unpacking. That bootstrap applied the new startup resources; then the ordinary internal-app-JAR startup worked normally.

[royteeuwen]

@krystian-panek-vmltech did you have time yet to check my contribution ? I'm happy to make any changes required

[krystian-panek-vmltech]

Tbh I have been working on ACM (https://github.com/wttech/acm) and Comparador (https://github.com/wttech/comparador) recently. Sorry but it's kinda volunteer-like job (not too much time to spend)😀 next time I will do open source it will be AEMC. I promise 😀