eseiler · Nov 5, 2026 · Nov 5, 2026 · Nov 5, 2026 · Nov 5, 2026 · Nov 5, 2026
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -35,7 +35,7 @@ jobs:
 
       - name: Build image
         working-directory: ./docker
-        run: docker build . &> ${{ runner.temp }}/build-${{ github.run_id }}.log
+        run: docker compose build &> ${{ runner.temp }}/build-${{ github.run_id }}.log
 
       - name: Upload log
         if: always()

diff --git a/README.md b/README.md
@@ -1,33 +1,62 @@
 Usage Statistics of OpenMS and SeqAn
 ====================================
 
-This repo contains scripts, data and other files, that are helpful for acquiring and analysing the usage statistics for OpenMS and SeqAn.
+This repo contains scripts, data and other files that create the usage statistics dashboards of OpenMS and SeqAn.
+A dashboard is a single HTML report that combines:
+
+* calls to the version check server: calls and users per month, top applications, operating systems, world map
+* downloads from GitHub releases, conda and PyPI
+* GitHub issues and pull requests
+* Google Scholar citations
 
 A cron job which triggers the analysis of log files is set up on [Jenkins](https://abibuilder.cs.uni-tuebingen.de/jenkins/job/usage_statistics/job/usageStatistics/).
 
 :exclamation:**Important**:exclamation:: The top level `seqan_versions.txt` and `openms_versions.txt` must not be moved to a different location because it is needed to ensure a correct server response to a user call. The Tübingen REST server pulls both file once per night, so if a version changes, they can be directly updated in this repository.
 
-How the scripts are used
-------------------------
-
-1. Get log files:
-   The original server calls are stored in Tuebingen.
-   Ask someone in Tuebingen to send you a zipped file or get access to the server yourself.
+Creating a dashboard
+--------------------
 
-2. Prepare log files:
-   The log files are originally not in a nice format to process.
-   You need to use the script `process_seqan_log_files.sh` or `process_openms_log_files.sh` first.
-   ```
-   $ /path/to/script/process_seqan_log_files.sh seqan-all.log
+1. Get the dependencies:
+   They are available as a Docker image, see [docker/README.md](docker/README.md).
+   ```console
+   $ docker pull ghcr.io/seqan/usage-plots:latest
    ```
-   Now the directory should have a file `all.log` and a file `geolocations.csv`
-   Those are the files you need for generating a plots and stuff
+   Without Docker, you need `gawk`, `perl`, `pandoc`, the Python packages from `scripts/config/requirements.txt` and the R packages from `scripts/config/r-requirements.txt`.
 
-3. Generate the user statistics report pdf:
-   To generate the pdf file execute the R script `create_report.R` and supply the according files.
-   Note: you will also get a new directory in your current path with all the single figures.
+2. Get log files:
+   The original server calls are stored in Tübingen.
+   Ask someone in Tübingen to send you a zipped file or get access to the server yourself.
+   Concatenate the logs into one file.
+   Its name has to start with `seqan-` or `openms-`, which selects the project.
+   To try things out, use the randomized log in [example/](example/).
+
+3. Set a GitHub token:
+   Without it, GitHub statistics are not collected, see [scripts/README.md](scripts/README.md#github-token).
+   ```console
+   $ export GITHUB_PAT="github_pat_[...]"
    ```
-   $ /path/to/script/create_report.R all.log geolocations.csv report.pdf
+
+4. Run `scripts/run.sh`:
+   It calls the individual steps sequentially and creates all output in the current working directory.
+   ```console
+   $ mkdir -p local/report && cd local/report # `local` is ignored by git
+   $ ../../scripts/run.sh ../../example/seqan-example.log
    ```
-   Note: The database file for ip geo locations was obtained from : https://lite.ip2location.com.
-   The pyhton code to access the database binary was obtained from: https://github.com/chrislim2888/IP2Location-Python
+   `output_seqan/report.html` is the dashboard.
+
+[example/output_seqan/](example/output_seqan/) contains the result of these commands.
+
+To change which GitHub repositories and conda/PyPI packages are counted as libraries or apps, edit `scripts/config/config.py`.
+[scripts/README.md](scripts/README.md) describes the configuration, the individual steps and the output.
+
+Repository layout
+-----------------
+
+| Path                                        | Content                                                                 |
+|---------------------------------------------|-------------------------------------------------------------------------|
+| `scripts/`                                  | Scripts that create the dashboard, see [README](scripts/README.md)      |
+| `docker/`                                   | Docker image with all dependencies, see [README](docker/README.md)      |
+| `example/`                                  | Randomized example log and its dashboard, see [README](example/README.md) |
+| `database/`                                 | IP geolocation database, obtained from https://lite.ip2location.com     |
+| `tooltags/`                                 | Tags for the tools of each project                                      |
+| `seqan_versions.txt`, `openms_versions.txt` | Current versions, pulled by the REST server (do not move)               |
diff --git a/docker/README.md b/docker/README.md
@@ -1,3 +1,51 @@
-This Dockerfile creates a slave that has all packages and R libs to create the usage statistics reports.
-To be run on abibuilder3 (or any other machine with access to the docker registry).
-For general testing, just replace the base container to an official Ubuntu image. E.g. ubuntu_1804 to ubuntu:18.04
+Docker image
+============
+
+The `Dockerfile` creates an image with all system, R and Python packages needed to create the usage statistics reports.
+The image contains only these dependencies, not the scripts of this repository.
+
+Getting the image
+-----------------
+
+The image is published as [ghcr.io/seqan/usage-plots](https://github.com/orgs/seqan/packages/container/package/usage-plots):
+
+```console
+$ docker pull ghcr.io/seqan/usage-plots:latest
+```
+
+Running the scripts
+-------------------
+
+From the root of a checkout, mount the repository and pass on the GitHub token (see [scripts/README.md](../scripts/README.md#github-token)).
+The container runs as user `jenkins`, which needs write access to the directory the output is created in.
+
+```console
+$ docker run --rm -it -e GITHUB_PAT -v "$PWD":/usage_plots -w /usage_plots ghcr.io/seqan/usage-plots:latest
+```
+
+This opens a shell in the container:
+
+```console
+$ mkdir -p local/report && cd local/report
+$ ../../scripts/run.sh ../../example/seqan-example.log
+```
+
+Building the image
+------------------
+
+The upload is not automated by CI yet; a new image has to be built and pushed manually.
+
+The `Dockerfile` copies the requirement files from `scripts/config` via an additional build context.
+`compose.yaml` defines this context, so build with `docker compose` instead of a plain `docker build`:
+
+```console
+$ docker compose -f docker/compose.yaml build
+$ docker push ghcr.io/seqan/usage-plots:latest
+```
+
+Base image
+----------
+
+The `Dockerfile` uses a Jenkins base image because the reports are generated by a Jenkins cron job.
+The scripts do not depend on it.
+To use a different base, change `FROM` and, if the new base has no `jenkins` user, `ARG user` in the `Dockerfile`.
diff --git a/example/README.md b/example/README.md
@@ -0,0 +1,41 @@
+Example
+=======
+
+A small example of the input and output of [`scripts/run.sh`](../scripts/README.md).
+
+| Path                | Content                                                  |
+|---------------------|----------------------------------------------------------|
+| `seqan-example.log` | 10,000 lines of the SeqAn version check log              |
+| `output_seqan/`     | Output for this log, `output_seqan/report.html` is the dashboard |
+
+To view the dashboard, check out the repository and open `output_seqan/report.html` in a browser.
+
+The log is randomized
+---------------------
+
+The log is based on real data, but its IP addresses and timestamps were replaced by random values.
+Locations, dates and trends in the report therefore do **not** reflect real usage:
+
+* Each IP address was replaced by a random, public address.
+* Each day was replaced by a random day of the same time span, and each call got a random time of day.
+* The timestamps embedded in the `test_version_check` requests match the new time of their line.
+
+Calls from the same day are still on the same day, and addresses from the same `/16` network are still in the same `/16` network.
+This keeps the merging of calls and the detection of clusters meaningful, see [Log processing](../scripts/README.md#log-processing).
+
+The remainder of each line is unchanged, and the applications and versions in the report come from real calls.
+
+The conda, PyPI, GitHub and Google Scholar statistics in the report do not depend on the log.
+They are real values from the date in the file names.
+
+Recreating the output
+---------------------
+
+From the root of the repository:
+
+```console
+$ mkdir -p local/report && cd local/report
+$ ../../scripts/run.sh ../../example/seqan-example.log
+```
+
+Creating the output in `example/` itself would add files for the current date next to the existing ones.
diff --git a/example/output_seqan/country_and_city_distribution_per_app_2026-11-05.tsv b/example/output_seqan/country_and_city_distribution_per_app_2026-11-05.tsv
@@ -0,0 +1,80 @@
+"Application"	"Countries"	"Cities"	"Country_with_most_calls"	"Calls_from_Country"	"City_with_most_calls"	"Calls_from_City"
+"razers3"	51	284	"United States of America"	847	"Ashburn"	174
+"flexbar"	24	44	"United States of America"	44	"Columbus"	25
+"GenMap"	19	38	"Japan"	24	"Tokyo"	24
+"yara_indexer"	18	35	"Japan"	66	"Tokyo"	64
+"yara_mapper"	14	41	"United States of America"	85	"Tokyo"	65
+"RLM"	13	23	"United States of America"	15	"Tokyo"	7
+"adaptFinder"	11	22	"United States of America"	9	"Tempe"	3
+"mason_simulator"	10	26	"United States of America"	19	"Shanghai"	3
+"mason_genome"	7	11	"United States of America"	5	"Beijing"	2
+"alf"	4	5	"United States of America"	5	"North Bergen"	4
+"gustaf_mate_joining"	4	5	"United States of America"	4	"North Bergen"	3
+"mason_variator"	4	9	"United States of America"	6	"Amsterdam"	1
+"seqan_tcoffee"	4	6	"United States of America"	3	"Fontana Liri"	1
+"Indexer"	3	3	"United States of America"	2	"North Bergen"	2
+"Penguin_Parade"	3	3	"United States of America"	3	"North Bergen"	3
+"Raptor"	3	6	"United States of America"	21	"North Bergen"	18
+"Test"	3	3	"Switzerland"	2	"Zurich"	2
+"compute_gain"	3	3	"Finland"	1	"Amsterdam"	1
+"dfi"	3	3	"United States of America"	2	"North Bergen"	2
+"fiona_illumina"	3	3	"Denmark"	1	"Amsterdam"	1
+"fx_bam_coverage"	3	3	"Brazil"	1	"Franca"	1
+"lambda"	3	6	"United States of America"	5	"Honolulu"	2
+"lambda2"	3	3	"Colombia"	1	"Bucaramanga"	1
+"lambda3"	3	3	"Switzerland"	16	"Zurich"	16
+"lambda_indexer"	3	5	"United States of America"	5	"Honolulu"	3
+"needle"	3	4	"Spain"	194	"Vigo"	194
+"ovgraphbuild"	3	4	"United States of America"	6	"Detroit"	5
+"rabema_prepare_sam"	3	3	"Italy"	1	"Amsterdam"	1
+"roi_feature_projection"	3	4	"Brazil"	2	"Al Mazzunah"	1
+"samcat"	3	3	"Denmark"	1	"Frederiksberg"	1
+"stellar"	3	6	"United States of America"	7	"North Bergen"	4
+"fiona"	2	6	"United States of America"	6	"Charlotte"	2
+"gustaf"	2	2	"Finland"	1	"Espoo"	1
+"insegt"	2	5	"United States of America"	5	"North Bergen"	2
+"lara"	2	2	"China"	2	"Wuhan"	2
+"my_program"	2	2	"Switzerland"	3	"North Bergen"	3
+"mygit"	2	2	"Netherlands (Kingdom of the)"	2	"Amsterdam"	2
+"pair_align"	2	2	"Indonesia"	1	"Mulia"	1
+"rabema_evaluate"	2	3	"United States of America"	2	"Chita"	1
+"sak"	2	2	"China"	1	"Beijing"	1
+"sam2matrix"	2	5	"United States of America"	3	"Columbus"	1
+"slimm"	2	2	"Netherlands (Kingdom of the)"	2	"Nijmegen"	2
+"tree_recon"	2	3	"United States of America"	4	"North Bergen"	3
+"AnnotateBam"	1	1	"China"	2	"Hangzhou"	2
+"Contamination"	1	1	"China"	2	"Hangzhou"	2
+"Demultiplexer"	1	1	"Malaysia"	1	"Kampung Kadok"	1
+"HmnTrimmer"	1	1	"Hong Kong"	1	"Hong Kong"	1
+"Mapper"	1	1	"Switzerland"	1	"Zurich"	1
+"MergeBam"	1	1	"China"	2	"Hangzhou"	2
+"MinionReadAnalyzer"	1	1	"China"	2	"Shanghai"	2
+"Search"	1	1	"Singapore"	1	"Singapore"	1
+"bam2roi"	1	1	"United States of America"	1	"North Bergen"	1
+"bisar"	1	1	"United States of America"	1	"San Francisco"	1
+"casbar"	1	2	"United States of America"	2	"Madison"	1
+"corrprofile"	1	1	"United States of America"	1	"Columbus"	1
+"count"	1	1	"New Zealand"	1	"Christchurch"	1
+"customgenomegenerator"	1	1	"Switzerland"	2	"Zurich"	2
+"four2three"	1	1	"United States of America"	1	"North Bergen"	1
+"fx_fastq_stats"	1	2	"United States of America"	2	"Ashburn"	1
+"ibf_bitscore"	1	1	"Argentina"	1	"Salta"	1
+"join"	1	1	"United States of America"	1	"North Bergen"	1
+"kgl_phylo"	1	1	"United States of America"	1	"Little Rock"	1
+"mason_frag_sequencing"	1	1	"United States of America"	1	"North Bergen"	1
+"mason_materializer"	1	2	"United States of America"	2	"Bellevue"	1
+"mason_methylation"	1	1	"Korea (the Republic of)"	1	"Seongnam"	1
+"mason_splicing"	1	1	"United States of America"	1	"North Bergen"	1
+"mhc"	1	1	"Taiwan (Province of China)"	1	"Taipei"	1
+"micro_razers"	1	1	"United States of America"	1	"North Bergen"	1
+"minions"	1	1	"Switzerland"	16	"Zurich"	16
+"pizzly"	1	1	"United States of America"	1	"Honolulu"	1
+"rabema_build_gold_standard"	1	2	"United States of America"	2	"Cupertino"	1
+"razers"	1	1	"United States of America"	1	"North Bergen"	1
+"select"	1	1	"New Zealand"	1	"Christchurch"	1
+"singleCellPipe"	1	1	"United States of America"	1	"Tampa"	1
+"slimm_build"	1	1	"United States of America"	1	"North Bergen"	1
+"snp_store"	1	3	"United States of America"	4	"North Bergen"	2
+"st_index_search"	1	2	"United States of America"	3	"North Bergen"	2
+"umis_per_barcode"	1	1	"New Zealand"	1	"Christchurch"	1
+"valik"	1	1	"United States of America"	2	"Atlanta"	2
diff --git a/example/output_seqan/daily_and_weekly_usage_per_app_2026-11-05.tsv b/example/output_seqan/daily_and_weekly_usage_per_app_2026-11-05.tsv
@@ -0,0 +1,80 @@
+"Application"	"Calls"	"Average_per_Day"	"Average_per_Week"
+"razers3"	2147	1.19344080044469	8.35408560311284
+"yara_mapper"	240	0.133407448582546	0.933852140077821
+"needle"	199	0.110617009449694	0.77431906614786
+"yara_indexer"	192	0.106725958866037	0.747081712062257
+"flexbar"	93	0.0516953863257365	0.361867704280156
+"GenMap"	78	0.0433574207893274	0.303501945525292
+"RLM"	36	0.0200111172873819	0.140077821011673
+"mason_simulator"	31	0.0172317954419122	0.120622568093385
+"adaptFinder"	26	0.0144524735964425	0.101167315175097
+"Raptor"	24	0.0133407448582546	0.0933852140077821
+"lambda3"	19	0.0105614230127849	0.0739299610894942
+"minions"	16	0.00889382990550306	0.0622568093385214
+"mason_genome"	12	0.00667037242912729	0.0466926070038911
+"ovgraphbuild"	12	0.00667037242912729	0.0466926070038911
+"mason_variator"	9	0.00500277932184547	0.0350194552529183
+"stellar"	9	0.00500277932184547	0.0350194552529183
+"alf"	8	0.00444691495275153	0.0311284046692607
+"lambda_indexer"	8	0.00444691495275153	0.0311284046692607
+"fiona"	7	0.00389105058365759	0.0272373540856031
+"gustaf_mate_joining"	7	0.00389105058365759	0.0272373540856031
+"lambda"	7	0.00389105058365759	0.0272373540856031
+"insegt"	6	0.00333518621456365	0.0233463035019455
+"my_program"	6	0.00333518621456365	0.0233463035019455
+"seqan_tcoffee"	6	0.00333518621456365	0.0233463035019455
+"Penguin_Parade"	5	0.00277932184546971	0.0194552529182879
+"sam2matrix"	5	0.00277932184546971	0.0194552529182879
+"tree_recon"	5	0.00277932184546971	0.0194552529182879
+"Indexer"	4	0.00222345747637576	0.0155642023346304
+"Test"	4	0.00222345747637576	0.0155642023346304
+"dfi"	4	0.00222345747637576	0.0155642023346304
+"mygit"	4	0.00222345747637576	0.0155642023346304
+"roi_feature_projection"	4	0.00222345747637576	0.0155642023346304
+"snp_store"	4	0.00222345747637576	0.0155642023346304
+"compute_gain"	3	0.00166759310728182	0.0116731517509728
+"fiona_illumina"	3	0.00166759310728182	0.0116731517509728
+"fx_bam_coverage"	3	0.00166759310728182	0.0116731517509728
+"lambda2"	3	0.00166759310728182	0.0116731517509728
+"lara"	3	0.00166759310728182	0.0116731517509728
+"rabema_evaluate"	3	0.00166759310728182	0.0116731517509728
+"rabema_prepare_sam"	3	0.00166759310728182	0.0116731517509728
+"samcat"	3	0.00166759310728182	0.0116731517509728
+"slimm"	3	0.00166759310728182	0.0116731517509728
+"st_index_search"	3	0.00166759310728182	0.0116731517509728
+"AnnotateBam"	2	0.00111172873818788	0.00778210116731518
+"Contamination"	2	0.00111172873818788	0.00778210116731518
+"MergeBam"	2	0.00111172873818788	0.00778210116731518
+"MinionReadAnalyzer"	2	0.00111172873818788	0.00778210116731518
+"casbar"	2	0.00111172873818788	0.00778210116731518
+"customgenomegenerator"	2	0.00111172873818788	0.00778210116731518
+"fx_fastq_stats"	2	0.00111172873818788	0.00778210116731518
+"gustaf"	2	0.00111172873818788	0.00778210116731518
+"mason_materializer"	2	0.00111172873818788	0.00778210116731518
+"pair_align"	2	0.00111172873818788	0.00778210116731518
+"rabema_build_gold_standard"	2	0.00111172873818788	0.00778210116731518
+"sak"	2	0.00111172873818788	0.00778210116731518
+"valik"	2	0.00111172873818788	0.00778210116731518
+"Demultiplexer"	1	0.000555864369093941	0.00389105058365759
+"HmnTrimmer"	1	0.000555864369093941	0.00389105058365759
+"Mapper"	1	0.000555864369093941	0.00389105058365759
+"Search"	1	0.000555864369093941	0.00389105058365759
+"bam2roi"	1	0.000555864369093941	0.00389105058365759
+"bisar"	1	0.000555864369093941	0.00389105058365759
+"corrprofile"	1	0.000555864369093941	0.00389105058365759
+"count"	1	0.000555864369093941	0.00389105058365759
+"four2three"	1	0.000555864369093941	0.00389105058365759
+"ibf_bitscore"	1	0.000555864369093941	0.00389105058365759
+"join"	1	0.000555864369093941	0.00389105058365759
+"kgl_phylo"	1	0.000555864369093941	0.00389105058365759
+"mason_frag_sequencing"	1	0.000555864369093941	0.00389105058365759
+"mason_methylation"	1	0.000555864369093941	0.00389105058365759
+"mason_splicing"	1	0.000555864369093941	0.00389105058365759
+"mhc"	1	0.000555864369093941	0.00389105058365759
+"micro_razers"	1	0.000555864369093941	0.00389105058365759
+"pizzly"	1	0.000555864369093941	0.00389105058365759
+"razers"	1	0.000555864369093941	0.00389105058365759
+"select"	1	0.000555864369093941	0.00389105058365759
+"singleCellPipe"	1	0.000555864369093941	0.00389105058365759
+"slimm_build"	1	0.000555864369093941	0.00389105058365759
+"umis_per_barcode"	1	0.000555864369093941	0.00389105058365759
diff --git a/example/output_seqan/downloads_apps_2026-11-05.tsv b/example/output_seqan/downloads_apps_2026-11-05.tsv
@@ -0,0 +1,4 @@
+"Start"	"End"	"GitHub_Downloads"	"Conda_Downloads"	"PyPI_Downloads"	"Total_Downloads"
+"2025 Q1"	"2025 Q4"	6	161128	0	161134
+"2025 Q4"	"2026 Q3"	0	97116	0	97116
+"2026 Q1"	"2026 Q4"	0	58988	0	58988
diff --git a/example/output_seqan/downloads_libs_2026-11-05.tsv b/example/output_seqan/downloads_libs_2026-11-05.tsv
@@ -0,0 +1,4 @@
+"Start"	"End"	"GitHub_Downloads"	"Conda_Downloads"	"PyPI_Downloads"	"Total_Downloads"
+"2025 Q1"	"2025 Q4"	943	19593	0	20536
+"2025 Q4"	"2026 Q3"	7918	16309	0	24227
+"2026 Q1"	"2026 Q4"	7984	11258	0	19242