Skip to content

perf(file-list): db pagination - #17505

Open
alperozturk96 wants to merge 1 commit into
masterfrom
fix/large-folder-listing
Open

perf(file-list): db pagination#17505
alperozturk96 wants to merge 1 commit into
masterfrom
fix/large-folder-listing

Conversation

@alperozturk96

Copy link
Copy Markdown
Collaborator

Issue

When user have a lot of files and/or folder in directory app crashes. Due to fetching all files at once from DB.

{
      "header": {
        "logLevel": "ERROR",
        "pid": 7829,
        "tid": 7849,
        "applicationId": "com.nextcloud.client",
        "processName": "com.nextcloud.client",
        "tag": "CursorWindow",
        "timestamp": {
          "seconds": 1787142876,
          "nanos": 504903577
        }
      },
      "message": "Failed to read row 615, column 5 from a window with 615 rows, 52 columns"
},

Changes

  • Fetch batch by batch. Batch size is 100.
  • Return ID from DB instead of FileEntity.

How to reproduce crash?

  1. Have 2K files and 2K folders in root directory.
  2. Keep scroll
  3. Crash

Signed-off-by: alperozturk96 <alper_ozturk@proton.me>
@alperozturk96 alperozturk96 added this to the Nextcloud App 35.0.0 milestone Aug 19, 2026
@alperozturk96 alperozturk96 added 3. to review performance 馃殌 Performance improvement opportunities (non-crash related) labels Aug 19, 2026
@github-actions

Copy link
Copy Markdown

blue-Light-Screenshot test failed, but no output was generated. Maybe a preliminary stage failed.

@github-actions

Copy link
Copy Markdown

Codacy

SpotBugs

CategoryBaseNew
Bad practice3232
Correctness5959
Dodgy code207207
Experimental11
Malicious code vulnerability22
Multithreaded correctness2525
Performance3939
Security1919
Total384384

@github-actions

Copy link
Copy Markdown

APK file: https://github.com/nextcloud/android/actions/runs/32258040777/artifacts/9367607142
To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.
qrcode (please click on link to get QR code displayed)

@daniele-verducci daniele-verducci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Alper,
I tried it, but it's still crashing for me. Here's the stacktrace:

2026-08-24 15:25:28.892 12717-12765 ReadFolder...eOperation com.nextcloud.android.qa             E  Synchronized /: failed
2026-08-24 15:25:28.893 12717-12743 WM-WorkerWrapper        com.nextcloud.android.qa             E  Work [ id=d7d541fe-9e26-4998-9176-87424ecf4e04, tags={ com.nextcloud.client.jobs.metadata.MetadataWorker,*,name:metadata_sync,timestamp:1787577910001,class:MetadataWorker } ] failed because it threw an exception/error
                                                                                                    java.lang.OutOfMemoryError: Failed to allocate a 16 byte allocation with 2295520 free bytes and 2241KB until OOM, target footprint 603979776, growth limit 603979776; giving up on allocation because <1% of heap free after GC.
                                                                                                    	at org.apache.harmony.xml.dom.InnerNodeImpl.getChildNodes(InnerNodeImpl.java:56)
                                                                                                    	at org.apache.jackrabbit.webdav.xml.DomUtil.getContent(DomUtil.java:397)
                                                                                                    	at org.apache.jackrabbit.webdav.property.DefaultDavProperty.createFromXml(DefaultDavProperty.java:130)
                                                                                                    	at org.apache.jackrabbit.webdav.MultiStatusResponse.createFromXml(MultiStatusResponse.java:463)
                                                                                                    	at org.apache.jackrabbit.webdav.MultiStatus.createFromXml(MultiStatus.java:185)
                                                                                                    	at org.apache.jackrabbit.webdav.client.methods.DavMethodBase.processResponseBody(DavMethodBase.java:282)
                                                                                                    	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1749)
                                                                                                    	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
                                                                                                    	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
                                                                                                    	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
                                                                                                    	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
                                                                                                    	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
                                                                                                    	at com.owncloud.android.lib.common.OwnCloudClient.executeMethod(OwnCloudClient.java:192)
                                                                                                    	at com.owncloud.android.lib.resources.files.ReadFolderRemoteOperation.run(ReadFolderRemoteOperation.java:62)
                                                                                                    	at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:193)
                                                                                                    	at com.owncloud.android.operations.RefreshFolderOperation.fetchAndSyncRemoteFolder(RefreshFolderOperation.java:466)
                                                                                                    	at com.owncloud.android.operations.RefreshFolderOperation.run(RefreshFolderOperation.java:281)
                                                                                                    	at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:132)
                                                                                                    	at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:141)
                                                                                                    	at com.nextcloud.client.jobs.metadata.MetadataWorker$refreshFolder$2.invokeSuspend(MetadataWorker.kt:108)
                                                                                                    	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
                                                                                                    	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
                                                                                                    	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
                                                                                                    	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:798)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
2026-08-24 15:25:34.018 12717-13036 ReadFolder...eOperation com.nextcloud.android.qa             E  Synchronized /: failed
2026-08-24 15:25:34.018 12717-13036 AndroidRuntime          com.nextcloud.android.qa             E  FATAL EXCEPTION: Thread-17
                                                                                                    Process: com.nextcloud.android.qa, PID: 12717
                                                                                                    java.lang.OutOfMemoryError: Failed to allocate a 80032 byte allocation with 968416 free bytes and 945KB until OOM, target footprint 603979776, growth limit 603979776; giving up on allocation because <1% of heap free after GC.
                                                                                                    	at org.apache.jackrabbit.webdav.MultiStatus.getResponses(MultiStatus.java:128)
                                                                                                    	at com.owncloud.android.lib.resources.files.ReadFolderRemoteOperation.readData(ReadFolderRemoteOperation.java:133)
                                                                                                    	at com.owncloud.android.lib.resources.files.ReadFolderRemoteOperation.run(ReadFolderRemoteOperation.java:70)
                                                                                                    	at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:193)
                                                                                                    	at com.owncloud.android.operations.RefreshFolderOperation.fetchAndSyncRemoteFolder(RefreshFolderOperation.java:466)
                                                                                                    	at com.owncloud.android.operations.RefreshFolderOperation.run(RefreshFolderOperation.java:281)
                                                                                                    	at com.owncloud.android.lib.common.operations.RemoteOperation.run(RemoteOperation.java:387)
                                                                                                    	at java.lang.Thread.run(Thread.java:1571)

If it's relevant, here's how I tested it:

  • pawned docker container (github.com/tobiasKaminsky/nextcloud-client-testing-server)
  • created empty dirs and ampty files with bash for loop and touch (so the files are 0 bytes .txt files)
  • runned occ files:scan
  • opened in browser to check the APIs are not crashing (OK)
  • installed QA APK build from CI and opened: crashes after a while

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3. to review performance 馃殌 Performance improvement opportunities (non-crash related)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants