Enhance Metadata Export Subsystem, Part 1

.github/workflows/reviewdog_checkstyle.yml

@@ -12,7 +12,7 @@ jobs:
             - name: Checkout
               uses: actions/checkout@v7
             - name: Run check style
-              uses: nikitasavinov/checkstyle-action@master
+              uses: nikitasavinov/checkstyle-action@1.0.0
               with:
                   fail_on_error: true
                   reporter: github-pr-review

doc/release-notes/12686-enhance-export-subsys.md

@@ -0,0 +1,28 @@
+### More robust metadata export
+
+The metadata export subsystem was overhauled: exports are generated more reliably, failures are reported more clearly, and cached exports are kept fresh more consistently.
+
+### Better error reporting for failed exports
+
+- When a dataset is (re-)exported in multiple formats, a failure of one format no longer aborts the others.
+  The resulting error message now lists exactly which formats failed (and the log records more context, including the dataset and format involved).
+- Requesting an export in a format that Dataverse does not recognize (for example through a typo in the `exporter=` query parameter)
+  now returns a clear HTTP 400 (Bad Request) response with a readable error message, instead of an uninformative server error (HTTP 500) or forbidden (HTTP 403) response.
+- Generating an export for an unreleased (draft) version no longer holds the entire export in memory; it is streamed through a temporary file instead.
+  Temporary export files are now created with owner-only permissions rather than the system default.
+
+### Updates for system administrators
+
+- Export failures now log clearer warnings including the dataset and the format(s) affected.
+  Repeatedly failing I/O operations (such as reading and deleting cached exports) escalate their log level and report recovery, making intermittent storage problems easier to spot.
+- **Note:** Cached metadata exports generated with this or a previous release are stored under their legacy auxiliary object name (for example `export_ddi.cached`). 
+  The new code manages the newer, version-qualified names instead (for example `export_ddi_1.0.cached`) and does not delete the legacy ones yet.
+  They may linger in a dataset's auxiliary storage until manual cleanup or a full re-export removes them. Purging of legacy cache entries is tracked as a follow-up task.
+
+### Updates for developers
+
+The previous monolithic `ExportService` was replaced by a small set of well-separated beans (`ExportServiceBean`, `ExportPipelineBean`, `ExporterRegistryBean`).
+They are built on a new `ExportCache` abstraction, with a `StorageIO`-based implementation as default.
+A new `ExportSystemException` hierarchy (subclasses `InternalFailure` and `InvalidRequest`) is mapped to appropriate HTTP statuses by JAX-RS exception mappers.
+Export cache entries are now keyed by the dataset's version, and the code comes with extensive new unit tests.
+The old `io.gdcc.spi.export.ExportException` is no longer used on or across export subsystem boundaries.

doc/sphinx-guides/source/developers/coding-style.md

@@ -101,6 +101,15 @@ Generally speaking you should use `fine` for everything that you don't want to s
 
 When adding logging, do not simply add `System.out.println()` lines because the logging level cannot be controlled.
 
+#### Log Level Escalation
+
+When ignoring or handling errors, often a debug-level error message is created.
+Low occurences of such errors may not warrant a sysadmin's attention.
+On the other hand, when such errors are more frequently encountered, log level escalation might be required.
+
+The `edu.harvard.iq.dataverse.util.logging.FailureEscalation` utility class helps you achieve such escalation patterns.
+An example on how to use the class is provided in its Javadocs.
+
 ### Avoid Hard-Coding Strings (Use Constants)
 
 Special strings should be defined as public constants. For example, `DatasetFieldConstant.java` contains a field for "title" and it's used in many places in the code (try "Find Usages" in Netbeans). This is better than writing the string "title" in all those places.
@@ -119,6 +128,11 @@ If you just downloaded Netbeans and are using the out-of-the-box settings, you s
 
 If you know of a way to easily share Netbeans configuration across a team, please get in touch.
 
+### Temporary Files Security
+
+To avoid exposing potentially sensitive information in temporary files stored on the file system, use the
+`edu.harvard.iq.dataverse.util.SecureTempFiles` utility instead of Java mechanics as `Files.createTempFile` directly.
+
 ## Bash
 
 Generally, Google's Shell Style Guide at <https://google.github.io/styleguide/shell.xml> seems to have good advice.

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

@@ -1,5 +1,6 @@
 package edu.harvard.iq.dataverse;
 
+import edu.harvard.iq.dataverse.export.service.ExportSystemException;
 import edu.harvard.iq.dataverse.provenance.ProvPopupFragmentBean;
 import edu.harvard.iq.dataverse.api.AbstractApiBean;
 import edu.harvard.iq.dataverse.authorization.AuthenticationServiceBean;
@@ -38,12 +39,11 @@
 import edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.PublishDataverseCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.UpdateDatasetVersionCommand;
-import edu.harvard.iq.dataverse.export.ExportService;
 import edu.harvard.iq.dataverse.settings.FeatureFlags;
+import edu.harvard.iq.dataverse.export.service.ExportServiceBean;
+import edu.harvard.iq.dataverse.export.service.ExporterRegistryBean;
 import edu.harvard.iq.dataverse.util.cache.CacheFactoryBean;
 import edu.harvard.iq.dataverse.util.json.JsonUtil;
-import io.gdcc.spi.export.ExportException;
-import io.gdcc.spi.export.Exporter;
 import edu.harvard.iq.dataverse.ingest.IngestRequest;
 import edu.harvard.iq.dataverse.ingest.IngestServiceBean;
 import edu.harvard.iq.dataverse.license.LicenseServiceBean;
@@ -105,7 +105,6 @@
 import jakarta.faces.view.ViewScoped;
 import jakarta.inject.Inject;
 import jakarta.inject.Named;
-import jakarta.json.Json;
 import jakarta.json.JsonObject;
 import jakarta.json.JsonObjectBuilder;
 import jakarta.persistence.OptimisticLockException;
@@ -255,6 +254,10 @@ public enum DisplayMode {
     DvObjectServiceBean dvObjectService;
     @EJB
     CacheFactoryBean cacheFactory;
+    @EJB
+    ExportServiceBean exportService;
+    @EJB
+    ExporterRegistryBean exporterRegistryService;
     @Inject
     DataverseRequestServiceBean dvRequestService;
     @Inject
@@ -4898,31 +4901,18 @@ public String getTabularDataFileURL(Long fileid) {
     }
 
     public List< String[]> getExporters(){
-        List<String[]> retList = new ArrayList<>();
-        String myHostURL = getDataverseSiteUrl();
-        for (String [] provider : ExportService.getInstance().getExportersLabels() ){
-            String formatName = provider[1];
-            String formatDisplayName = provider[0];
-
-            Exporter exporter = null;
-            try {
-                exporter = ExportService.getInstance().getExporter(formatName);
-            } catch (ExportException ex) {
-                logger.warning("Failed to get : " + formatName);
-                logger.warning(ex.getLocalizedMessage());
-                exporter = null;
-            }
-            if (exporter != null && exporter.isAvailableToUsers()) {
-                // Not all metadata exports should be presented to the web users!
-                // Some are only for harvesting clients.
-
-                String[] temp = new String[2];
-                temp[0] = formatDisplayName;
-                temp[1] = myHostURL + "/api/datasets/export?exporter=" + formatName + "&persistentId=" + dataset.getGlobalId().asString();
-                retList.add(temp);
-            }
-        }
-        return retList;
+        String urlTemplate = getDataverseSiteUrl() + "/api/datasets/export?exporter=%s&persistentId=%s";
+
+        return exporterRegistryService.getDetails().stream()
+            .filter(ExporterRegistryBean.Details::isAvailableToUsers)
+            .map(details -> new String[]{
+                details.localizedDisplayName(),
+                urlTemplate.formatted(
+                    details.formatName(),
+                    dataset.getGlobalId().asString()
+                )
+            })
+            .toList();
     }
 
 
@@ -6136,9 +6126,15 @@ public String getCroissant() {
             // The full version is available from the "Export Metadata" dropdown.
             // Both versions are available via API.
             final String CROISSANT_SCHEMA_NAME = "croissantSlim";
-            ExportService instance = ExportService.getInstance();
-            String croissant = instance.getLatestPublishedAsString(dataset, CROISSANT_SCHEMA_NAME);
-            if (FeatureFlags.CROISSANT_WITH_LOCAL_REVIEWS.enabled()) {
+            String croissant = null;
+            try {
+                croissant = exportService.getLatestPublishedAsString(dataset, CROISSANT_SCHEMA_NAME);
+            } catch (ExportSystemException e) {
+                logger.log(Level.WARNING, "Unable to get latest published croissant schema as String", e);
+                return null;
+            }
+
+            if (croissant != null && FeatureFlags.CROISSANT_WITH_LOCAL_REVIEWS.enabled()) {
                 // Rewrite the export on the fly and insert local reviews until we have a solution for https://github.com/gdcc/dataverse-spi/issues/5
                 JsonObjectBuilder reviewsJsonObj = null;
                 try {
@@ -6169,8 +6165,7 @@ public List<License> getAvailableLicenses(){
 
     public String getJsonLd() {
         if (isThisLatestReleasedVersion()) {
-            ExportService instance = ExportService.getInstance();
-            String jsonLd = instance.getLatestPublishedAsString(dataset, SchemaDotOrgExporter.NAME);
+            String jsonLd = exportService.getLatestPublishedAsString(dataset, SchemaDotOrgExporter.NAME);
             if (jsonLd != null) {
                 logger.fine("Returning cached schema.org JSON-LD.");
                 return jsonLd;
@@ -7019,7 +7014,7 @@ public String getSignpostingLinkHeader() {
             return null;
         }
         if (signpostingLinkHeader == null) {
-            SignpostingResources sr = new SignpostingResources(systemConfig, workingVersion,
+            SignpostingResources sr = new SignpostingResources(systemConfig, exporterRegistryService, workingVersion,
                     JvmSettings.SIGNPOSTING_LEVEL1_AUTHOR_LIMIT.lookupOptional().orElse(""),
                     JvmSettings.SIGNPOSTING_LEVEL1_ITEM_LIMIT.lookupOptional().orElse(""));
             signpostingLinkHeader = sr.getLinks();

src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java

@@ -5,7 +5,7 @@
 import edu.harvard.iq.dataverse.authorization.Permission;
 import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser;
 import edu.harvard.iq.dataverse.authorization.users.User;
 import edu.harvard.iq.dataverse.dataaccess.DataAccess;
 import edu.harvard.iq.dataverse.dataaccess.ImageThumbConverter;
 import edu.harvard.iq.dataverse.dataaccess.StorageIO;
 import edu.harvard.iq.dataverse.dataset.DatasetUtil;
@@ -16,7 +16,8 @@
 import edu.harvard.iq.dataverse.engine.command.impl.DestroyDatasetCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.FinalizeDatasetPublicationCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.GetDatasetStorageSizeCommand;
-import edu.harvard.iq.dataverse.export.ExportService;
+import edu.harvard.iq.dataverse.export.service.ExportServiceBean;
+import edu.harvard.iq.dataverse.export.service.ExportSystemException;
 import edu.harvard.iq.dataverse.globus.GlobusServiceBean;
 import edu.harvard.iq.dataverse.harvest.server.OAIRecordServiceBean;
 import edu.harvard.iq.dataverse.pidproviders.FailedPIDResolutionLoggingServiceBean;
@@ -103,6 +104,9 @@
 
     @EJB
     UserNotificationServiceBean userNotificationService;
+
+    @EJB
+    ExportServiceBean exportService;
 
     private static final SimpleDateFormat logFormatter = new SimpleDateFormat("yyyy-MM-dd'T'HH-mm-ss");
 
@@ -1079,17 +1083,11 @@
 
         // and finally,
         if (countCachedExtras) {
-            // count the sizes of the files cached for the dataset itself
-            // (i.e., the metadata exports):
-            StorageIO<Dataset> datasetSIO = DataAccess.getStorageIO(dataset);
-
-            for (String[] exportProvider : ExportService.getInstance().getExportersLabels()) {
-                String exportLabel = "export_" + exportProvider[1] + ".cached";
-                try {
-                    total += datasetSIO.getAuxObjectSize(exportLabel);
-                } catch (IOException ioex) {
-                    // safe to ignore; object not cached
-                }
+            try {
+                // Count the sizes of the files cached for the dataset itself (i.e., the metadata exports):
+                total += exportService.usedCacheStorage(dataset);
+            } catch (IOException | ExportSystemException ex) {
+                logger.log(Level.INFO, ex, () -> "Failed to count cached metadata exports size for the dataset: " + dataset.getId() + " (ignoring)");
             }
         }
 

src/main/java/edu/harvard/iq/dataverse/EjbDataverseEngine.java

@@ -6,6 +6,8 @@
 import edu.harvard.iq.dataverse.authorization.AuthenticationServiceBean;
 import edu.harvard.iq.dataverse.authorization.providers.builtin.BuiltinUserServiceBean;
 import edu.harvard.iq.dataverse.dataverse.featured.DataverseFeaturedItemServiceBean;
+import edu.harvard.iq.dataverse.export.service.ExportServiceBean;
+import edu.harvard.iq.dataverse.export.service.ExporterRegistryBean;
 import edu.harvard.iq.dataverse.license.LicenseServiceBean;
 import edu.harvard.iq.dataverse.util.cache.CacheFactoryBean;
 import edu.harvard.iq.dataverse.engine.DataverseEngine;
@@ -209,6 +211,12 @@ public class EjbDataverseEngine {
     @EJB
     CacheFactoryBean cacheFactory;
 
+    @EJB
+    ExportServiceBean exportService;
+
+    @EJB
+    ExporterRegistryBean exporterRegistry;
+
     @Resource
     EJBContext ejbCtxt;
 
@@ -664,7 +672,17 @@ public MetadataBlockServiceBean metadataBlocks() {
                 public DatasetTypeServiceBean datasetTypes() {
                     return datasetTypeService;
                 }
-
+
+                @Override
+                public ExportServiceBean exportService() {
+                    return exportService;
+                }
+
+                @Override
+                public ExporterRegistryBean exporterRegistry() {
+                    return exporterRegistry;
+                }
+
                 @Override
                 public void beginCommandSequence() {
                     this.commandsCalled = new Stack();

src/main/java/edu/harvard/iq/dataverse/FilePage.java

@@ -24,9 +24,10 @@
 import edu.harvard.iq.dataverse.engine.command.impl.RestrictFileCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.UningestFileCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.UpdateDatasetVersionCommand;
-import edu.harvard.iq.dataverse.export.ExportService;
-import io.gdcc.spi.export.ExportException;
-import io.gdcc.spi.export.Exporter;
+import edu.harvard.iq.dataverse.export.service.ExportServiceBean;
+import edu.harvard.iq.dataverse.export.service.ExportSystemException;
+import edu.harvard.iq.dataverse.export.service.ExporterRegistryBean;
+import edu.harvard.iq.dataverse.export.service.ExporterRegistryBean.Details;
 import edu.harvard.iq.dataverse.externaltools.ExternalTool;
 import edu.harvard.iq.dataverse.externaltools.ExternalToolHandler;
 import edu.harvard.iq.dataverse.externaltools.ExternalToolServiceBean;
@@ -36,7 +37,6 @@
 import edu.harvard.iq.dataverse.makedatacount.MakeDataCountLoggingServiceBean;
 import edu.harvard.iq.dataverse.makedatacount.MakeDataCountLoggingServiceBean.MakeDataCountEntry;
 import edu.harvard.iq.dataverse.privateurl.PrivateUrlServiceBean;
-import edu.harvard.iq.dataverse.settings.FeatureFlags;
 import edu.harvard.iq.dataverse.settings.JvmSettings;
 import edu.harvard.iq.dataverse.settings.SettingsServiceBean;
 import edu.harvard.iq.dataverse.util.BundleUtil;
@@ -64,7 +64,6 @@
 import jakarta.faces.application.FacesMessage;
 import jakarta.faces.component.UIComponent;
 import jakarta.faces.context.FacesContext;
-import jakarta.faces.validator.ValidatorException;
 import jakarta.faces.view.ViewScoped;
 import jakarta.inject.Inject;
 import jakarta.inject.Named;
@@ -129,6 +128,10 @@ public class FilePage implements java.io.Serializable {
     IngestServiceBean ingestService;
     @EJB
     SystemConfig systemConfig;
+    @EJB
+    ExportServiceBean exportService;
+    @EJB
+    ExporterRegistryBean exporterRegistryService;
 
 
     @Inject
@@ -466,30 +469,19 @@ public void setVersion(String version) {
         this.version = version;
     }
 
-    public List< String[]> getExporters(){
-        List<String[]> retList = new ArrayList<>();
-        String myHostURL = systemConfig.getDataverseSiteUrl();
-        for (String [] provider : ExportService.getInstance().getExportersLabels() ){
-            String formatName = provider[1];
-            String formatDisplayName = provider[0];
-
-            Exporter exporter = null; 
-            try {
-                exporter = ExportService.getInstance().getExporter(formatName);
-            } catch (ExportException ex) {
-                exporter = null;
-            }
-            if (exporter != null && exporter.isAvailableToUsers()) {
-                // Not all metadata exports should be presented to the web users!
-                // Some are only for harvesting clients.
-
-                String[] temp = new String[2];
-                temp[0] = formatDisplayName;
-                temp[1] = myHostURL + "/api/datasets/export?exporter=" + formatName + "&persistentId=" + fileMetadata.getDatasetVersion().getDataset().getGlobalId().asString();
-                retList.add(temp);
-            }
-        }
-        return retList;  
+    public List<String[]> getExporters(){
+        String urlTemplate = systemConfig.getDataverseSiteUrl() + "/api/datasets/export?exporter=%s&persistentId=%s";
+
+        return exporterRegistryService.getDetails().stream()
+            .filter(Details::isAvailableToUsers)
+            .map(details -> new String[]{
+                details.localizedDisplayName(),
+                urlTemplate.formatted(
+                    details.formatName(),
+                    fileMetadata.getDatasetVersion().getDataset().getGlobalId().asString()
+                )
+            })
+            .toList();
     }
 
     public String saveProvFreeform(String freeformTextInput, DataFile dataFileFromPopup) throws CommandException {
@@ -640,15 +632,13 @@ public String uningestFile() throws CommandException {
         editDataset = file.getOwner();
         if (editDataset.isReleased()) {
             try {
-                ExportService instance = ExportService.getInstance();
-                instance.exportAllFormats(editDataset);
-
-            } catch (ExportException ex) {
+                exportService.exportAllFormats(editDataset);
+            } catch (ExportSystemException ex) {
                 // Something went wrong!
                 // Just like with indexing, a failure to export is not a fatal
                 // condition. We'll just log the error as a warning and keep
                 // going:
-                logger.log(Level.WARNING, "Uningest: Exception while exporting:{0}", ex.getMessage());
+                logger.log(Level.WARNING, ex, () -> "Uningest: Exception while exporting " + editDataset.getId());
             }
         }
         datafileService.save(file);

src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java

@@ -22,6 +22,8 @@
 import edu.harvard.iq.dataverse.engine.command.impl.GetLatestAccessibleDatasetVersionCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.GetLatestPublishedDatasetVersionCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.GetSpecificPublishedDatasetVersionCommand;
+import edu.harvard.iq.dataverse.export.service.ExportServiceBean;
+import edu.harvard.iq.dataverse.export.service.ExporterRegistryBean;
 import edu.harvard.iq.dataverse.externaltools.ExternalToolServiceBean;
 import edu.harvard.iq.dataverse.license.LicenseServiceBean;
 import edu.harvard.iq.dataverse.makedatacount.DatasetMetricsServiceBean;
@@ -246,6 +248,12 @@ String getWrappedMessageWhenJson() {
 
     @EJB
     TemplateServiceBean templateSvc;
+
+    @EJB
+    ExportServiceBean exportSvc;
+
+    @EJB
+    ExporterRegistryBean exporterRegistrySvc;
 
     @Inject
     FailedPIDResolutionLoggingServiceBean fprLogService;