Provide option for configuring `publishAllChecksums` when using `com.gradleup.nmcp` plugin

[marcphilipp]

What feature do you want to see added?

When using the com.gradleup.nmcp plugin, there's currently no way to configure publishAllChecksums to be true because it's hard-coded as false:

nmcp/nmcp/src/main/kotlin/nmcp/internal/DefaultNmcpExtension.kt

Line 81 in 92e0f77

publishAllChecksums = project.provider { false }

Could you please add an option to NmcpExtension for configuring it?

Upstream changes

Recent introduction of checksum filtering

Are you interested in contributing this feature?

Yes, I could do that if you agree this is the right solution.

[martinbonnin]

Thanks for opening this and apologies for breaking your workflow as I tend to focus on the aggregation case.

2 questions:

• why do you need the checksums? I'm genuinely curious since Gradle almost never uses it (I think there's an optimization case but it sounds quite niche). I was hoping long term we could deprecate publishAllChecksums
• If you really need them, could you use both com.gradleup.nmcp and com.gradleup.nmcp.aggregation? The option is not hardcoded for the aggregation.

plugins {
  id("com.gradleup.nmcp") // for exposing all the publications
  id("com.gradleup.nmcp.aggregation") // for configuring the aggregation
}

nmcpAggregation {
  centralPortal {
    username.set("")
    password.get("")
    publishAllChecksums.set(true)
  }
}

dependencies {
  nmcpAggregation(project)
}

[marcphilipp]

Thanks for the quick reply! 👍

why do you need the checksums?

Since sha1 and md5 are weak/insecure, we'd like to keep publishing sha256/sha512 checksums. While one should rather verify the signatures to ensure files have not been tampered with, some folks might use the checksums for that. In addition, JUnit has users who don't use Gradle/Maven but download and verify jars directly. They also could be relying on those to be present.

I think filtering out checksums for signature files makes sense, though. Gradle also seems to be moving in that direction (see gradle/gradle#20232 (comment)). For the main JUnit repo, I've now enabled publishAllChecksums and filtered out those myself (see junit-team/junit-framework@673d68f). Would be great if that were an option of the nmcp plugins directly.

If you really need them, could you use both com.gradleup.nmcp and com.gradleup.nmcp.aggregation? The option is not hardcoded for the aggregation.

I suppose then I might as well switch to the settings plugin, right? In any case, I'd then have to use the publishAggregationToCentralPortal task instead of publishAllPublicationsToCentralPortal, correct? I was trying to keep things simple for single-project repos such as https://github.com/junit-team/testng-engine.

[martinbonnin]

While one should rather verify the signatures to ensure files have not been tampered with, some folks might use the checksums for that. In addition, JUnit has users who don't use Gradle/Maven but download and verify jars directly. They also could be relying on those to be present.

Is there evidence of people doing this? Since the files are next to each other, I'm not sure this gives any amount of security? If someone can tamper with the .jar files, surely, they can also tamper with its checksum?

For the main JUnit repo, I've now enabled publishAllChecksums and filtered out those myself (see junit-team/junit-framework@673d68f). Would be great if that were an option of the nmcp plugins directly.

Let's see where the community lands on this. My hunch is that sending and storing checksums in Central puts additional stress on the shared infrastructure for very little (if any) benefit and we should all be better not publishing those.

I also have an ongoing discussion with @brianf to remove the requirement to publish checksums altogether (including md5 and sha1). Maybe the conclusion is that we actually need those but so far I don't think anyone has a compelling use case.

I suppose then I might as well switch to the settings plugin, right? In any case, I'd then have to use the publishAggregationToCentralPortal task instead of publishAllPublicationsToCentralPortal, correct? I was trying to keep things simple for single-project repos such as https://github.com/junit-team/testng-engine.

Right 👍 . The settings plugin is the easiest entry point for most users. It wasn't a thing initially but I'd like to steer usage that direction.

[marcphilipp]

Is there evidence of people doing this?

I don't recall the specifics but I remember someone doing that on our issue tracker a while ago.

Since the files are next to each other, I'm not sure this gives any amount of security? If someone can tamper with the .jar files, surely, they can also tamper with its checksum?

Sure, that's true.

My hunch is that sending and storing checksums in Central puts additional stress on the shared infrastructure for very little (if any) benefit and we should all be better not publishing those.

I think they served a purpose originally when uploading a new release to ensure integrity of files. I'd be interested in the outcome of your discussion. 🙂

I don't know how much stress the small checksum files put on the system, but storage-wise surely other files are more heavy. We looked at our biggest files and found Javadoc jars to be the culprit. Each contained copies of the fonts that are used. We've now removed those which resulted in a drop from 69 MB to 12 MB. @sormuras is trying to address this issue on the JDK level so the default is more suited for publication.

The settings plugin is the easiest entry point for most users

I'll give that a try on those repos. As I'm already using it for our multi-project repos, I'm sure it'll work fine. If you don't want to address this use case for the standalone com.gradleup.nmcp plugin, feel free to close this issue.

[martinbonnin]

S3 has a minimum object size of 128kB so those small files could add up. They also clutter the browsing of the repositories and encourage (IMO) outdated practices. As it stands, I see very little reason to keep uploading them. If you bump into someone using them, please let me know, I'm curious the use case.

I'll give that a try on those repos. As I'm already using it for our multi-project repos, I'm sure it'll work fine. If you don't want to address this use case for the standalone com.gradleup.nmcp plugin, feel free to close this issue.

Thanks for understanding. I have deprecated publishAllPublicationsToCentralPortal in #264 and will focus on aggregation/settings plugins moving forward.

[leonard84]

Since the files are next to each other, I'm not sure this gives any amount of security? If someone can tamper with the .jar files, surely, they can also tamper with its checksum?

Don't forget that there are third-party proxies as well, e.g., Artifactory.

When investigating a possible breach, or for integrity monitoring, downloading a checksum file is way less traffic than re-downloading the original binary to verify integrity. AFAIK, there is no metadata protocol for retrieving the checksum from Maven Central, so plain files it is.

If you bump into someone using them, please let me know, I'm curious the use case.

The build tools use them to verify the file integrity. Although Gradle currently uses .sha1. A switch to sha256/sha512 would be possible in the future, but only if people keep using it.
md5/sha1 are deprecated for any security-relevant use case by RFC 9155 (also see cited sources, such as NIST). Changing a standard, unfortunately, takes a long time, but if we now stop publishing more secure checksum files, we are actively working against it.

If we want a compromise, I'd say drop sha256 and keep only the most secure, sha512.
And drop the hashes from signature files; they are indeed redundant.

[martinbonnin]

When investigating a possible breach, or for integrity monitoring, downloading a checksum file is way less traffic than re-downloading the original binary to verify integrity.

That makes sense 👍
Interestingly, I was just (as of 10 min ago) pointed at gradle/gradle#38329 which is the same thing (but with ~/.m2 being used as a cache).

AFAIK, there is no metadata protocol for retrieving the checksum from Maven Central, so plain files it is.

That's a bit sad. We could also think of putting the checksum in the pom or Gradle metadata or so... But problem for another day!

e.g., Artifactory.

Do we know if artifactory supports validating using .sha512? If yes, that's a good reason to only require one checksum (and probably .sha512)

If we want a compromise, I'd say drop sha256 and keep only the most secure, sha512.

Unrelated to this issue but I'd like to drop .md5 and .sha1 as well. I get that Central is adding quotas but making OSS developers pay for the rest of the ecosystem still being stuck on .md5 feels wrong.