diff --git a/docs/README.md b/docs/README.md index c450d8522..7dc214b79 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,8 @@ -# CogStack NiFi Documentation +# CogStack NiFi documentation -The central documentation for cogstack, hosted on docs.cogstack.org +The published documentation is hosted at +[cogstack-nifi.readthedocs.io](https://cogstack-nifi.readthedocs.io/en/latest/). +Run the commands below from the repository root. ## Setup @@ -12,9 +14,7 @@ The central documentation for cogstack, hosted on docs.cogstack.org ### Installation ```bash -uv venv --python 3.12 --allow-existing -source .venv/bin/activate -uv sync --dev +uv sync --project docs --frozen ``` ## Usage @@ -24,17 +24,24 @@ uv sync --dev To preview the documentation locally with live reload: ```bash -uv run mkdocs serve +uv run --project docs mkdocs serve ``` -The documentation will be available at `http://127.0.0.1:8000` +The documentation will be available at `http://127.0.0.1:8000`. ### Build documentation To build the static site: ```bash -uv run mkdocs build +uv run --project docs mkdocs build --strict ``` -The built site will be in the `site/` directory. +The built site will be in the repository's `site/` directory. + +Run the documentation-specific checks with: + +```bash +python3 scripts/tests/lint_markdown.py +python3 scripts/tests/check_docs_links.py +``` diff --git a/docs/_static/img/legacy_images/cogstack_common_schema_full_workflow.png b/docs/_static/img/legacy_images/cogstack_common_schema_full_workflow.png deleted file mode 100644 index 98b6520b7..000000000 Binary files a/docs/_static/img/legacy_images/cogstack_common_schema_full_workflow.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/configure-db-connector-w1.png b/docs/_static/img/legacy_images/configure-db-connector-w1.png deleted file mode 100644 index 4d8d4a601..000000000 Binary files a/docs/_static/img/legacy_images/configure-db-connector-w1.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/configure-db-reader-w1.png b/docs/_static/img/legacy_images/configure-db-reader-w1.png deleted file mode 100644 index 02bec34aa..000000000 Binary files a/docs/_static/img/legacy_images/configure-db-reader-w1.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/configure-db-reader-w2.png b/docs/_static/img/legacy_images/configure-db-reader-w2.png deleted file mode 100644 index d06d09539..000000000 Binary files a/docs/_static/img/legacy_images/configure-db-reader-w2.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/configure-es-reader-w4.png b/docs/_static/img/legacy_images/configure-es-reader-w4.png deleted file mode 100644 index cb2a7f6af..000000000 Binary files a/docs/_static/img/legacy_images/configure-es-reader-w4.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/configure-es-writer-w1.png b/docs/_static/img/legacy_images/configure-es-writer-w1.png deleted file mode 100644 index f6db595ee..000000000 Binary files a/docs/_static/img/legacy_images/configure-es-writer-w1.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/configure-medcat-w3.png b/docs/_static/img/legacy_images/configure-medcat-w3.png deleted file mode 100644 index 52e635c31..000000000 Binary files a/docs/_static/img/legacy_images/configure-medcat-w3.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/configure-tika-w2.png b/docs/_static/img/legacy_images/configure-tika-w2.png deleted file mode 100644 index bfd373fb3..000000000 Binary files a/docs/_static/img/legacy_images/configure-tika-w2.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/nifi-exec-workflow-w1.png b/docs/_static/img/legacy_images/nifi-exec-workflow-w1.png deleted file mode 100644 index b50fa1f74..000000000 Binary files a/docs/_static/img/legacy_images/nifi-exec-workflow-w1.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/nifi-templates-w1.png b/docs/_static/img/legacy_images/nifi-templates-w1.png deleted file mode 100644 index d5a2608b4..000000000 Binary files a/docs/_static/img/legacy_images/nifi-templates-w1.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/nifi-workflow-w2.png b/docs/_static/img/legacy_images/nifi-workflow-w2.png deleted file mode 100644 index 2c08a6af1..000000000 Binary files a/docs/_static/img/legacy_images/nifi-workflow-w2.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/nifi-workflow-w3.png b/docs/_static/img/legacy_images/nifi-workflow-w3.png deleted file mode 100644 index 1c1766d8c..000000000 Binary files a/docs/_static/img/legacy_images/nifi-workflow-w3.png and /dev/null differ diff --git a/docs/_static/img/legacy_images/nifi-workflow-w4.png b/docs/_static/img/legacy_images/nifi-workflow-w4.png deleted file mode 100644 index 15c6c101d..000000000 Binary files a/docs/_static/img/legacy_images/nifi-workflow-w4.png and /dev/null differ diff --git a/docs/deploy/configuration.md b/docs/deploy/configuration.md index 7f1adc10d..c4240be58 100644 --- a/docs/deploy/configuration.md +++ b/docs/deploy/configuration.md @@ -1,83 +1,123 @@ +# βš™οΈ Configuration +CogStack separates deployment configuration from local-development security +settings: +- `deploy/*.env` configures services, images, ports, resource limits, and the + shared Docker network. +- `security/env/*.env` configures certificate generation and local-development + users. +- Standalone services under `services/` keep additional environment files in + their own directories. -## Environment variables +The tracked values are development defaults. Keep deployment-specific values +outside the repository or inject them with your platform's secret manager. +Never commit production credentials or generated private keys. -As mentioned above, environment variables have been made available after release 1.0. -The variables are configurable, and are separated, into security and general env vars, furthermore, all services declared in the `services.yml` file have their variables in separate files. -In most cases, modifying these variables should be the only thing that is needed in order to run a successful deployment. +## Core deployment files -Multiple files are available, split into two categories: +The core Compose file, `deploy/services.yml`, loads these files: -- Service configuration under `deploy/`. -- Certificate and local-development credential configuration under `security/env/`. +| File | Main purpose | +|------|--------------| +| `deploy/project.env` | Compose project naming and shared project settings | +| `deploy/general.env` | General Docker logging and shared defaults | +| `deploy/nifi.env` | NiFi image, ports, resources, proxy settings, and paths | +| `deploy/elasticsearch.env` | Elasticsearch/OpenSearch, Dashboards/Kibana, and Beats | +| `deploy/database.env` | Sample and production database resources | +| `deploy/gitea.env` | Gitea image, ports, and application settings | +| `deploy/nginx.env` | NiFi nginx proxy settings | +| `deploy/telemetry.env` | Telemetry configuration | +| `deploy/network_settings.env` | Hostnames and shared network settings | -The variables declared in the `./deploy` folder are used in multiple config files, as follows: -- `elasticsearch.env`, variables here are used in : - - `./services/elasticsearch/config/(opensearch|elasticsearch).yml` - - `./services/kibana/config/(opensearch|elasticsearch).yml` - - `./services/metricbeat/metricbeat.yml` - - `./deploy/services.yml` in the following sections: `nifi`, `elasticsearch-1`, `elasticsearch-1`, `elasticsearch-3`, `kibana`, `metricbeat-1`,`metricbeat-2` +The following security files are also loaded where required: -- `nifi.env`, vars used in: - - `./deploy/services.yml`, sections: `nifi` - - `./nifi/conf/nifi.properties` +- `security/env/certificates_general.env` +- `security/env/certificates_elasticsearch.env` +- `security/env/certificates_nifi.env` +- `security/env/users_database.env` +- `security/env/users_elasticsearch.env` +- `security/env/users_nginx.env` +- `security/env/users_nifi.env` -- `jupyter.env`, vars used in: - - `./deploy/services.yml`, sections: `jupyter` +See the `env_file` anchors at the top of `deploy/services.yml` for the exact +files consumed by each core service. -- `nlp_service.env`, vars used in: - - `./deploy/services.yml`, sections: `nlp-medcat-service-production` +## Standalone service files -- `database.env`, vars used in: - - `./deploy/services.yml`, sections: `cogstack-databank-db`, `samples-db` +Services launched from their own Compose projects use environment files below +their respective directories. The deployment helper currently loads, when +present: -- `general.env`, these vars are optional, declared any custom variables you want here, used in the `nifi` section +- `services/cogstack-jupyter-hub/env/jupyter.env` +- `services/ocr-service/env/ocr_service.env` +- `services/cogstack-nlp/medcat-service/env/app.env` +- `services/cogstack-nlp/medcat-service/env/medcat.env` +- `services/cogstack-nlp/medcat-trainer/envs/env-prod` -Additional environment files under `security/env/` configure certificate generation and local accounts: +Check each service's Compose file for variant-specific files, such as MedCAT +de-identification or OCR text-only overrides. -- `certificates_elasticsearch.env` -- `certificates_general.env` -- `certificates_nifi.env` -- `users_database.env` -- `users_elasticsearch.env` -- `users_nginx.env` -- `users_nifi.env` +## Loading configuration +Make targets load the required files automatically. If you invoke Docker +Compose directly, change to the deployment directory and load the same +environment first: -### Customization -The tracked `.env` values are public development defaults. For a real deployment, keep deployment-specific copies outside the repository or inject the values through the platform's secret-management mechanism. Never commit production credentials. +```bash +cd deploy +source ./export_env_vars.sh +``` + +To inspect the combined environment used by the Makefile: + +```bash +make -C deploy show-env +``` + +Be careful when sharing this output because it includes development +credentials and certificate passwords. + +## Deployment-specific overrides + +For a separate deployment, copy the tracked defaults to a protected location +and edit the copies there: ```bash cp deploy/*.env /path/to/deployment-config/ cp security/env/*.env /path/to/deployment-config/security/ ``` -### Multiple deployments on the same machine -When deploying multiple docker-compose projects on the same machine (e.g. for dev or testing), it can be useful to remove all containers, volume and network names from the docker-compose file, and let [Docker create names](https://docs.docker.com/compose/reference/envvars/#compose_project_name) based on `COMPOSE_PROJECT_NAME` in `deploy/.env`. Docker will automatically create a Docker network and makes sure that containers can find each other by container name. +You must then adapt your Compose invocation or deployment tooling to load those +copies. The repository Makefile always loads the paths listed in +`deploy/export_env_vars.sh`. -For example, when setting `COMPOSE_PROJECT_NAME=cogstack-prod`, Docker Compose will create a container named `cogstack-prod_elasticsearch-1_1` for the `elasticsearch-1` service. Within the NiFi container, which is running in the same Docker network, you can refer to that container using just the service name `elasticsearch-1`. +## Multiple deployments on one host -
+The supplied Compose file uses explicit container and volume names in several +places. Merely changing `COMPOSE_PROJECT_NAME` therefore does not isolate every +resource. For concurrent deployments, use deployment-specific Compose +overrides that remove or replace explicit names and assign distinct host ports. -## Important security detail +Within a Compose network, services should address one another by service name, +for example `elasticsearch-1`, rather than by host-published ports. -Generated certificates and private keys are stored under `security/certificates/` and ignored by Git. Create them with `make -C deploy init-security`. Do not reuse development certificates or public development credential defaults in production. +## Security setup -## Services -Please note that all the services are deployed using [Docker](https://docker.io) engine and requires docker deamon to be running / functioning. +Generated certificates and private keys are stored under +`security/certificates/` and ignored by Git. Generate the local certificate sets +before starting services: -Please see [the available services](./services.md) for more details. +```bash +make -C deploy init-security +``` +Do not reuse development certificates or tracked development credentials in +production. See the [security overview](../security/main.md) for production +guidance. ## Workflows -Apache NiFi provides users the ability to build very large and complex data flows. -These data flows can be later saved as workflow *templates*, exported into XML format and shared with other users. -We provide few example templates for ingesting the records from a database into Elasticsearch and to perform extraction of NLP annotations from documents. - -### Deployment using Makefile -For deployments based on example workflows, please see [current workflows](./workflows.md) -and [legacy workflows](./workflows_legacy.md) for more details. -### Deployment using a custom Docker-compose -When using a fork of this repository for a customized deployments, it can be useful to copy `services.yml` to a deployment-specific `docker-compose.yml`. In this Compose file you can specify the services you need for your instance and configure all parameters per service, as well as track this file in a branch in your own fork. This way you can use your own version control and rebase on `CogStack/CogStack-NiFi` master without running into merge conflicts. +For settings required by the supplied NiFi templates, see the +[workflow documentation](./workflows.md). For general service endpoints and +ports, see [Services](./services.md). diff --git a/docs/deploy/deployment.md b/docs/deploy/deployment.md index 32ac6b428..6875c6379 100644 --- a/docs/deploy/deployment.md +++ b/docs/deploy/deployment.md @@ -15,12 +15,15 @@ Make sure you have read the [Prerequisites](./main.md) section before proceeding - environment variables that apply **only to the services defined inside `services.yml`**. - Security-related `.env` files (certificates, users) are under **`/security`** - These variables configure NiFi, Elasticsearch/OpenSearch, Kibana, Jupyter, Metricbeat, the sample DB, etc. + These variables configure core services such as NiFi, Elasticsearch/OpenSearch, + OpenSearch Dashboards or Kibana, Beats, and the databases. -> **Important:** If you run `docker compose` directly (instead of `make`), first load the envs with: +> **Important:** If you run `docker compose` directly (instead of `make`), +> change to the deployment directory and load the environment first: > > ```bash -> source ./deploy/export_env_vars.sh +> cd deploy +> source ./export_env_vars.sh > ``` > > The Makefile targets already do this for you. @@ -168,9 +171,20 @@ You can optionally create `gitea-admin-credentials` with `username` and `passwor ## 🧰 Makefile Command Overview -A concise reference for controlling the full CogStack deployment stack (NiFi, Elasticsearch, JupyterHub, MedCAT, OCR-service, GitEA, Beats, DB, etc.). +A concise reference for controlling the CogStack deployment stack (NiFi, +Elasticsearch, MedCAT, OCR, Gitea, Beats, databases, and optional services). All commands automatically load environment variables via `export_env_vars.sh`. +Before running a `start-*` target on a fresh checkout, generate the local +certificate sets: + +```bash +make -C deploy init-security +``` + +The command is idempotent and leaves complete existing certificate sets +unchanged. + ### πŸ”Ž Discover available Make targets You can list all available `deploy/Makefile` targets with descriptions: @@ -273,7 +287,7 @@ make -C deploy remote-delete-service \ | `make start-git-ea` | Start GitEA | | `make start-production-db` | Start Databank DB | | **`make start-data-infra`** | Start NiFi + Elastic + Samples DB | -| **`make start-all`** | Full stack: data infra + NLP + JupyterHub + OCR | +| **`make start-all`** | Start data infrastructure + MedCAT inference + both OCR services | --- @@ -298,7 +312,7 @@ make -C deploy remote-delete-service \ | `make stop-git-ea` | Stop GitEA | | `make stop-production-db` | Stop Databank DB | | **`make stop-data-infra`** | Stop NiFi + Elastic + Samples | -| **`make stop-all`** | Stop entire stack | +| **`make stop-all`** | Stop the service groups wired into the `stop-all` target | --- @@ -338,43 +352,20 @@ make -C deploy remote-delete-service \ ### πŸ“ Notes -- All `start-*` commands use `docker compose -f services.yml` unless referencing a specific service’s Dockerfile. -- `start-all` and `stop-all` act as the top-level orchestration entry points. -- Environment variables are **always sourced** using the integrated `WITH_ENV` macro. - ---- - -If you want, I can also generate a **minimal cheat sheet**, or an **ASCII tree diagram** that shows how `start-all` expands into all services. - -## πŸš€ Starting the Services - -All core services defined in `services.yml` can be started using the Makefile in the `deploy/` directory. - -Before running any `start-*` command on a fresh checkout, generate the -deployment-local certificates: - -```bash -make init-security -``` - -This command is idempotent and leaves complete existing certificate sets -unchanged. The examples below assume that you are in the `deploy/` directory. - -For most services in the `services` folder that are not part of the core stack defined in `services.yml` and are pulled from external git submodule repositories, the start-up process is the same. - -### ▢️ Start each service individually +- Core-service targets use `deploy/services.yml`; standalone-service targets + use the Compose files in their respective `services/` directories. +- `start-all` currently starts `start-data-infra`, `start-medcat-service`, and + `start-ocr-services`. Start other optional services explicitly. +- `stop-all` follows the dependencies declared in the Makefile; check + `make -C deploy help` before relying on it for a customised deployment. +- Environment variables are sourced using the integrated `WITH_ENV` macro. -You can start individual components of the CogStack-NiFi stack using the `make start-*` commands. -Each target loads all required environment variables automatically via `export_env_vars.sh`. +## πŸš€ Common startup patterns -This is useful for: - -- debugging a single service -- restarting only one component after config changes -- running lightweight subsets of the stack -- isolating problems or logs per service - ---- +The commands below assume that you are in `deploy/`. From the repository root, +use the same target with `make -C deploy`, for example +`make -C deploy start-data-infra`. The command table above is the complete +target reference. ### 🧩 Core NiFi Services @@ -408,39 +399,31 @@ Ideal for running ingestion pipelines and ETL workflows. --- -#### πŸ›’οΈ Elasticsearch / OpenSearch Services +### πŸ›’οΈ Elasticsearch/OpenSearch and dashboards -Please note that to switch from OpenSearch (Amazon open-source fork) to ElasticSearch you will need to change some environment variables, see the [configuration](./configuration.md) section. +To switch between OpenSearch and Elasticsearch, update the backend variables +described in [Configuration](./configuration.md) before generating certificates +or starting the services. ```bash make start-elastic ``` -Starts the standard 2-node Elasticsearch cluster + Kibana. +Starts Elasticsearch/OpenSearch nodes 1 and 2 plus Kibana or OpenSearch +Dashboards. ```bash make start-elastic-cluster ``` -Starts all 3 ES nodes. Useful for testing clustering, sharding, and replication. - -```bash -make start-elastic-1 -make start-elastic-2 -make start-elastic-3 -``` - -Start individual Elasticsearch nodes for debugging or failure-scenario testing. - ---- - -#### πŸ“ˆ Kibana +Starts all three search nodes. Start the dashboard separately if needed: ```bash make start-kibana ``` -Starts Kibana for inspecting logs, checking index mappings, monitoring ES health, and debugging pipelines. +For debugging or failure testing, start an individual node with +`make start-elastic-1`, `make start-elastic-2`, or `make start-elastic-3`. --- @@ -450,15 +433,15 @@ Starts Kibana for inspecting logs, checking index mappings, monitoring ES health make start-samples ``` -Starts **samples-db**, the small example DB used for demo flows. +Starts `samples-db`, which contains the datasets used by the example database +workflows. ```bash make start-production-db ``` -Starts the **cogstack-databank-db** production database. - -Use when testing SQL ingestion or verifying DB-driven NiFi flows. +Starts the PostgreSQL `cogstack-databank-db` service for deployments using a +separate ingestion database. --- @@ -468,68 +451,72 @@ Use when testing SQL ingestion or verifying DB-driven NiFi flows. make start-jupyter ``` -Starts the CogStack JupyterHub instance. Used for notebooks, analysis, model testing, and visualisation. +Starts the JupyterHub service at `https://localhost:8888`. Its production +Compose configuration joins the external `cogstack-net` network, so start the +core data infrastructure first unless that network already exists. + +JupyterHub is not included in `start-all`; start it explicitly when notebooks +or interactive analysis are required. --- -### 🧠 NLP Services (MedCAT Service & Trainer) +### 🧠 MedCAT services ```bash make start-medcat-service ``` -Starts the MedCAT concept extraction inference API. +Starts the MedCAT concept-extraction API used by the annotation workflow. ```bash make start-medcat-service-deid ``` -Starts the MedCAT DEID (de-identification) inference API. +Starts the MedCAT de-identification API. ```bash make start-medcat-trainer ``` -Starts the full MedCAT Trainer stack (Trainer UI + Solr + NGINX). Useful for annotation and supervised training tasks. +Starts MedCAT Trainer, its nginx proxy, and Solr for annotation and supervised +training. --- -### πŸ“ OCR Services +### πŸ“ OCR services ```bash make start-ocr-services ``` -Starts: - -- **ocr-service** β€” main OCR pipeline -- **ocr-service-text-only** β€” lightweight OCR/text extraction - -Use for PDF ingestion, OCR debugging, and pipeline validation. +Starts both `ocr-service` and `ocr-service-text-only`. Use these services for +document extraction, OCR workflows, and pipeline testing. --- -### πŸ› οΈ Miscellaneous Services (GIT EA) +### πŸ—‚οΈ Gitea ```bash make start-git-ea ``` -Starts the internal Gitea Git server used for local code/config storage. +Starts the local Gitea service for internal repository and configuration +hosting. --- -### πŸš€ Start the Entire Stack +### πŸš€ Start the standard service set ```bash make start-all ``` -Starts everything: +Starts: - Core infra -- JupyterHub -- MedCAT NLP services -- OCR services +- MedCAT inference service +- OCR and text-only OCR services -Use for complete deployments, demos, or full-stack development. +It does not start every optional service. Start Gitea, the production database, +MedCAT de-identification, MedCAT Trainer, or other optional services with their +individual targets when needed. diff --git a/docs/deploy/main.md b/docs/deploy/main.md index da3d73760..38a87f7df 100755 --- a/docs/deploy/main.md +++ b/docs/deploy/main.md @@ -6,7 +6,7 @@ Please read carefully as there can be many points of failure when installing/dep Please note that the OSes mentioned below are the versions we support, whatever is not listed here is not supported, and we will not provide support for. -- Linux OS (Ubuntu 24.04 LTS +, Debian 10+ are preffered, RHEL 9+). +- Linux OS (Ubuntu 24.04 LTS+, Debian 10+ preferred, or RHEL 9+). - Windows 11+/Windows Server 2022+ (Requires [WSL 2.0](https://learn.microsoft.com/en-us/windows/wsl/about) installation and the installation of an Ubuntu image, for a working setup, follow [this guide](https://documentation.ubuntu.com/wsl/latest/howto/install-ubuntu-wsl2/) to get going, and get back here when things are working). - macOS 15+ (Sequoia). @@ -22,7 +22,7 @@ Software required on machine (the minimum/basics to get demos running): ## πŸ” Other requirements (User Permissions/Firewall) - a Linux account with 'admin' rights, if possible, if not, you will need to get your IT team to take a look at this README and install the packages for you using the steps below (make sure they look at [Docker rootless installation steps](https://docs.docker.com/engine/security/rootless)) - - firewall whitelisting of the following addreses: + - firewall allow-listing of the following addresses: - https://github.com/ - https://hub.docker.com/ - https://docker.io diff --git a/docs/deploy/services.md b/docs/deploy/services.md index 2b7d8e709..b490761d0 100644 --- a/docs/deploy/services.md +++ b/docs/deploy/services.md @@ -1,7 +1,8 @@ # 🧩 Services -This section provides a complete overview of all services included in the CogStack-NiFi deployment. -All services run in Docker and interact within a shared internal Docker network. +This section provides an overview of the services included in the CogStack-NiFi +deployment. Services run in Docker and are attached to the relevant Compose +network when they need to communicate. --- @@ -20,18 +21,18 @@ The core services defined in `services.yml` include: - **samples-db** β€” PostgreSQL database populated with demo datasets. - **cogstack-databank-db / cogstack-databank-db-mssql** β€” Production-grade PostgreSQL and optional MSSQL instances. - **elasticsearch-1 / elasticsearch-2 / elasticsearch-3** β€” Multi-node Elasticsearch or OpenSearch cluster. -- **metricbeat / filebeat** β€” Elastic monitoring and log forwarder services. +- **metricbeat / filebeat** β€” Elastic monitoring and log-forwarding services. - **nifi** β€” Apache NiFi single-node instance with embedded ZooKeeper. - **nifi-nginx** β€” Reverse proxy providing secure access to NiFi. -- **ocr-service / ocr-service-text-only** β€” High-performance Python OCR and text extraction services. -- **nlp-medcat-service-production** β€” MedCAT NLP model service with REST API. -- **medcat-trainer-ui / medcat-trainer-nginx** β€” Web UI and reverse proxy for model training and refinement. +- **kibana** β€” Kibana or OpenSearch Dashboards, depending on the backend. +- **gitea** β€” Self-hosted Git service. +- **cogstack-cohort** β€” Cohort discovery interface. -- **kibana** β€” OpenSearch Dashboards UI. -- **jupyter-hub** β€” Fully featured data science interface. -- **git-ea** β€” Self‑hosted Git service (Gitea). +Additional services, including MedCAT, MedCAT Trainer, OCR, and JupyterHub, are +started from Compose files in their respective `services/` directories. -> πŸ” **Note:** Important configuration options and environment variables for these services are managed in `services.yml` and the associated `.env` files under `deploy/` and `security/`. +> πŸ” **Note:** Core-service configuration is managed in `services.yml` and +> the associated environment files under `deploy/` and `security/env/`. ## πŸ—‚οΈ Service Definitions @@ -48,7 +49,7 @@ Some services expose ports to the host for convenience. ## πŸ—£οΈ NLP/OCR and other services API Endpoints -Most web ETL & data-enrichment API services that we use will offer thw following endpoints for querying. +Most web ETL and data-enrichment APIs provide some or all of these endpoints: - **GET** `/api/info` - **POST** `/api/process` @@ -60,17 +61,20 @@ Useful for NiFi workflows (see [workflows](./workflows.md)). ## 🧬 MedCAT Service -Runs a REST API for model inference uses the [MedCAT library](https://github.com/CogStack/cogstack-nlp/tree/main/medcat-v2) which performss clinical concept extraction and linking. +This REST API uses the +[MedCAT library](https://github.com/CogStack/cogstack-nlp/tree/main/medcat-v2) +for clinical concept extraction and linking. The service has two operation modes: -- concept detection: exctracts medical concepts: outputs original text + annotations list. -- de-id mode aka. AnonCAT mode, for de-identifying documents: outputs de-identified text + (will output annotations that represent what was de-id in a future version). +- **Concept detection** extracts medical concepts and returns the original text + with annotations. +- **De-identification (AnonCAT)** returns de-identified text. ### Access -- `https://localhost:5555/api/info` - NER container, check if model loads successfully -- `https://localhost:5556/api/info` - DE-ID/AnonCAT container +- `http://localhost:5555/api/info` β€” concept extraction service +- `http://localhost:5556/api/info` β€” de-identification service ### Containers @@ -79,16 +83,14 @@ The service has two operation modes: ### Service location & files -- dir: `/services/cogstack-nlp/medcat-service/` -- docker compose file: `/services/cogstack-nlp/medcat-service/docker/docker-compose.yml` +- directory: `services/cogstack-nlp/medcat-service/` +- Docker Compose file: `services/cogstack-nlp/medcat-service/docker/docker-compose.yml` - env: located in `services/cogstack-nlp/medcat-service/env/` - ```bash - app.env - controls APP settings (number of cpus used, log level, etc) used by the NER container cogstack-medcat-service-production - medcat.env - used by the NER container, controls MedCAT settings directly. - app_deid.env - used by the DE-ID container, same app setting control, the main difference being the `APP_DEID_MODE`. - medcat_deid.env - used by the DE-ID container, controls MedCAT settings directly - ``` + - `app.env` β€” application settings for the concept extraction service + - `medcat.env` β€” MedCAT settings for concept extraction + - `app_deid.env` β€” application settings for de-identification + - `medcat_deid.env` β€” MedCAT settings for de-identification ### Ports @@ -100,8 +102,8 @@ The service has two operation modes: ### Models - A default MedMentions `MedMen` NER+L model (includes MetaCAT models) is available for public use but needs to be downloaded. -- To download a model head to the directory of the service `services/cogstack-nlp/medcat-service/scripts` -- Execute: `bash download_medmen.sh`, wait for download to complete. +- To download it, go to `services/cogstack-nlp/medcat-service/scripts` and run + `bash download_medmen.sh`. ### README @@ -115,7 +117,7 @@ Provides UI workflows for annotation, correction, and iterative model training. ### Access -- `https://localhost:8001` +- `http://localhost:8001` ### Containers @@ -188,13 +190,13 @@ Demo dataset with: - cleaned reports - annotation tables -### Acess +### Access -- `localhost:5555` +- `localhost:5554` ### Ports -- external: `5432` +- external: `5554` - internal: `5432` ### Credentials @@ -214,8 +216,7 @@ Place schema files inside and they will be loaded instantly on container startup services/cogstack-db//schemas/ ``` -Where `` can be: `mssql`,`pgsql`. - +Where `` can be `mssql` or `pgsql`. ### Credentials PostgreSQL and MSSQL development credentials are defined in `security/env/users_database.env`. Production credentials must be injected through deployment-local configuration or a secret manager. @@ -223,12 +224,12 @@ PostgreSQL and MSSQL development credentials are defined in `security/env/users_ ### Access - PgSQL: `localhost:5558` β†’ container `5432` -- MSSQL: `localhost:1443` β†’ container `1433` +- MSSQL: `localhost:1433` β†’ container `1433` ### Containers -- PgSQL: `cogstack-databank-db` -- MSSQL: `cogstack-databank-db-mssql` +- PgSQL: `cogstack-production-databank-db` +- MSSQL: `cogstack-production-databank-db-mssql` ### Service location & files @@ -264,6 +265,7 @@ Single-user development credentials are defined in `security/env/users_nifi.env` ### Containers - NiFi: `cogstack-nifi` +- nginx: `cogstack-nifi-nginx` ### Service location & files @@ -276,9 +278,14 @@ Single-user development credentials are defined in `security/env/users_nifi.env` ### Ports -| Component | External Port | Internal Port | -|---------------------|---------------|----------------| -| NiFi | `8443` | `8082`, `10000` | +| Component | External port | Internal port | +|-----------|---------------|---------------| +| NiFi via nginx | `8443` | `8443` | +| NiFi direct access | `8082` | `8443` | + +The NiFi container also publishes its input socket on container port `10000` +without assigning a fixed host port. Use `docker compose port nifi 10000` from +the `deploy/` directory to see the assigned host port. --- @@ -290,7 +297,7 @@ This service is fully described in the Elasticsearch section of the documentatio The repo supports both: -- ElasticSearch (native) +- Elasticsearch (native) - OpenSearch (Amazon fork) Switch between modes via environment variables in `deploy/elasticsearch.env`. @@ -315,9 +322,10 @@ Development credentials for both backends are defined in `security/env/users_ela #### Ports -- all ports need to be exposed via firewall to allow for intercluster communication, we assume 1 different port per node if hosted on the same machine/VM, in production mode all machines can have and use the following ports (if they live on separarate VMs/machines ): `9200`, `9300`, `9600` -- internal: `9300`, `9301`, `9302`, `9600`, `9601`, `9602`, `9200`, `9201`, `9202` -- external: `9300`, `9301`, `9302`, `9600`, `9601`, `9602`, `9200`, `9201`, `9202` +The example Compose deployment assigns distinct host ports to nodes running on +the same machine. Do not expose transport or analyzer ports through a public +firewall. For a production multi-host cluster, allow only the required traffic +between trusted cluster nodes. | Node | HTTP | Transport | Analyzer | |------|------|-----------|----------| @@ -362,15 +370,15 @@ Both run as independent containers in the deployment. Metricbeat: -- `metricbeat-1` -- `metricbeat-2` -- `metricbeat-3` +- `cogstack-metricbeat-1` +- `cogstack-metricbeat-2` +- `cogstack-metricbeat-3` Filebeat: -- `filebeat-1` -- `filebeat-2` -- `filebeat-3` +- `cogstack-filebeat-1` +- `cogstack-filebeat-2` +- `cogstack-filebeat-3` #### **Service Location & Files** @@ -385,7 +393,7 @@ Filebeat: #### **Ports** No external ports exposed. -All communication occurs internally within the `cogstack-net` Docker network. +All communication occurs internally within the `cognet` Docker network. #### **Notes** @@ -461,8 +469,7 @@ Image selection controlled by: ## πŸ€– OCR Service -High-performance document text extraction engine replacing legacy Tika for OCR + text processing. -In the near future it will be possible to use LLMs/custom models for ocr-ing (pending v2 release, ETA 2026). +High-performance document text-extraction engine for OCR and text processing. The service comes in **two variants**: @@ -481,7 +488,7 @@ Both expose a simple REST API. ### Access - ocr-service: `http://localhost:8090/api/process` -- ocr-seervice-text-only: `http://localhost:8091/api/process` +- ocr-service-text-only: `http://localhost:8091/api/process` ### Containers @@ -496,7 +503,8 @@ cogstacksystems/cogstack-ocr-service: ### Service Location & Files -- docker compose file: `services/ocr-service/docker/docker-compose.yml` +- Docker Compose files: `services/ocr-service/docker/docker-compose.base.yml` + and `services/ocr-service/docker/docker-compose.prod.yml` - service directory: `services/ocr-service/` - logs: - Host: `services/ocr-service/log/` @@ -520,7 +528,7 @@ Please check the service's own [README.md](https://github.com/CogStack/ocr-servi --- -## πŸ—‚οΈ Git-ea +## πŸ—‚οΈ Gitea Self-hosted Git instance (Gitea). Lightweight GitHub/GitLab-style service used for hosting repositories inside secure or offline environments. @@ -537,7 +545,7 @@ Lightweight GitHub/GitLab-style service used for hosting repositories inside sec ### Containers -- `gitea` +- `cogstack-gitea` ### Service Location & Files @@ -551,7 +559,7 @@ Persistent repository data is stored in the volume defined in `services.yml`. | Service | External | Internal | |---------|----------|----------| -| Git-ea | `3000` | `3000` | +| Gitea | `3000` | `3000` | ### Notes @@ -563,51 +571,37 @@ Persistent repository data is stored in the volume defined in `services.yml`. ## 🧱 NGINX -*Note: this component may eventually be replaced by **Traefik** as the preferred reverse‑proxy and ingress layer for CogStack deployments.* - -NGINX is used as a lightweight reverse proxy to provide secure, unified access to internal CogStack services. -It handles HTTPS, routing, and access control for NiFi, MedCAT Trainer, and other components. - -MedCAT-Trainer has its own nginx instance that runs independently. +NGINX is used as a lightweight reverse proxy to provide HTTPS access to NiFi. +MedCAT Trainer has a separate nginx instance in its own Compose project. **Purpose:** -- Secure external access to internal services -- Reverse proxy for NiFi, MedCAT Trainer, and service UIs -- TLS termination (optional) -- Basic auth / access control where required - -Two variants are included: - -- **nginx-nifi** β€” main proxy for NiFi and related services -- **nginx-medcat-trainer** β€” specialized proxy for MedCAT Trainer - -Two variants: - -- **nginx-nifi** β€” main proxy for services -- **nginx-medcat-trainer** β€” dedicated trainer proxy +- **nifi-nginx** terminates TLS and proxies NiFi UI and API traffic. +- **medcattrainer_nginx** proxies the MedCAT Trainer application separately. ### Access Examples (actual paths depend on config): - NiFi: `https://localhost:8443` -- MedCAT Trainer: `https://localhost:8001` +- MedCAT Trainer: `http://localhost:8001` Routing rules are defined in the NGINX configuration files. ### Containers -- `nifi-nginx` β€” main proxy for NiFi -- `medcat-trainer-nginx` β€” proxy dedicated to MedCAT Trainer +- `cogstack-nifi-nginx` β€” proxy for NiFi +- `medcattrainer_nginx` β€” proxy for MedCAT Trainer ### Service Location & Files -- docker compose file: `deploy/services.yml`, trainer - `deploy/cogstack-nlp/medcat-trainer` +- NiFi Compose file: `deploy/services.yml` +- MedCAT Trainer Compose file: + `services/cogstack-nlp/medcat-trainer/docker-compose-prod.yml` - config files: - - `services/nginx/config/nifi.conf` - - `services/nginx/config/medcat-trainer.conf` - - additional templates under `services/nginx/config/` + - `services/nginx/config/nginx.conf.template` + - `services/cogstack-nlp/medcat-trainer/nginx/nginx.conf` + - `services/cogstack-nlp/medcat-trainer/nginx/sites-enabled/` - env / certificates: - `security/env/certificates_general.env` - `security/env/certificates_nifi.env` diff --git a/docs/deploy/troubleshooting.md b/docs/deploy/troubleshooting.md index 33bbe4177..66da5f5a8 100644 --- a/docs/deploy/troubleshooting.md +++ b/docs/deploy/troubleshooting.md @@ -1,127 +1,193 @@ # πŸ“› Troubleshooting -Always start with fresh containers and volumes, to make sure that there are no volumes from previous experimentations, make sure to always delete all/any cogstack running containers by executing: +Start with logs and container state. Do not delete containers or volumes as a +routine first step: volumes may contain NiFi state, database data, or search +indices. -`docker container rm samples-db elasticsearch-1 kibana nifi nlp-medcat-service-production tika-service nlp-gate-drugapp nlp-medcat-snomed nlp-gate-bioyodie medcat-trainer-ui medcat-trainer-nginx jupyter-hub -f` +From the repository root, list containers and their state: -followed by a cleanup or dangling volumes (careful as this will remove all volumes which are NOT being used by a container, if you want to remove specific volumes you will have to manually specifiy the volume names), otherwise, you can specify : +```bash +docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' +``` + +Inspect the affected service before restarting it: + +```bash +docker logs --tail 300 cogstack-nifi +docker inspect cogstack-nifi +``` -`docker volume prune -f` WARNING THIS WILL DELETE ALL UNUSED VOLUMES ON YOUR MACHINE!. Check the volume names used in services.yml file and delete them as necessary `dockr volume rm volume_name` +Use the scoped Make targets where possible. For example: -## 🐞 Known Issues/errors +```bash +make -C deploy stop-nifi +make -C deploy start-nifi +``` -Common issues that can be encountered across services. +To recreate only the NiFi containers while retaining named volumes: -### 🍎 **Apple Silicon** +```bash +make -C deploy delete-nifi-containers +make -C deploy start-nifi +``` -Many services cannot run natively on Apple Silicon (such as M1 and M2 architectures). Common error messages related to Apple silicon follow patterns similar to: -

- - `no match for platform in manifest` -

-

- - `no matching manifest for linux/arm64/v8 in the manifest list entries` -

-

- - `image with reference cogstacksystems/cogstack-ocr-service:1.0.2 was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64` -

-To solve these issues; Rosetta is required and enabled in Docker Desktop. Finally an environment variable is required to be set. +!!! danger "Deleting persistent data" -Rosetta can which can be installed via the following command: + `make -C deploy cleanup` runs `docker compose down -v` for the core + deployment and deletes its named volumes. Use it only when you deliberately + want to reset all core service data. Do not use a global + `docker volume prune` command as a CogStack troubleshooting step because it + can delete unrelated Docker volumes on the host. + +## 🐞 Known issues + +### 🍎 Apple Silicon + +Some images may only publish an `linux/amd64` variant. Typical errors include: + +```text +no match for platform in manifest +no matching manifest for linux/arm64/v8 in the manifest list entries +image ... does not match the specified platform: wanted linux/arm64, actual: linux/amd64 +``` + +On macOS, install Rosetta if it is not already available: ```bash softwareupdate --install-rosetta ``` -When Rosetta and Docker Desktop are installed, Rosetta must be enabled. This done by going to Docker Desktop -> Setting -> General and enabling "Use Virtualization framework". After in the same settings go to "features in development" -> "Use Rosetta for x86/amd64 emulation on Apple Silicon". Finally execute the following command: +Enable Docker Desktop's Apple Virtualization framework and Rosetta-based +`x86_64` emulation. If an image still needs an explicit platform, set it for the +current shell before starting the service: ```bash export DOCKER_DEFAULT_PLATFORM=linux/amd64 ``` -to set the environment variable. These issues are known to occur on the "cogstack-nifi", "cogstack-ocr-services" and "jupyter-hub" services and may occur on others. +Emulation is slower than running a native ARM64 image and may require higher +Docker Desktop memory limits. + +### πŸ”§ NiFi -### πŸ”§ **NiFi** +#### NiFi reports mixed HTTP and HTTPS configuration -When dealing with contaminated deployments ( containers using volumes from previous instances ) : -

- - `NiFi only supports one mode of HTTP or HTTPS operation...` deleting the volumes should usually solve this issue, if not, please check the `nifi.properties` if there have been modifications done by yourself or a developer on it. -

- - building the NiFi image manually on a restricted system, this is usually not necessary, but if for some reason this needs to be done then some settings such as proxy configs might need to be set up in the `nifi/Dockerfile` epecially ones related to the `grape` application and dealing with external downloads. -

- - `keystore.jks`/`truststore.jks` related errors, remove the nifi container & related volumes then restart the nifi instance. -

- - `HTTP 421 INVALID PROXY` or `System Error: Invalid host header` occurs when NiFi does not trust the public host/port forwarded by nginx. For remote deployments, set `NIFI_WEB_PROXY_HOST` in `deploy/nifi.env` to the exact browser-visible host and port, for example `localhost:8443,nifi.example.org:8443`, and keep `NIFI_WEB_PROXY_PORT` aligned with that public port. If the public URL is `https://nifi.example.org/nifi`, use port `443`; if it is `https://nifi.example.org:8443/nifi`, use port `8443`. Restart `nifi` and `nifi-nginx` afterwards. If the generated container config still shows the old value, check `/opt/nifi/nifi-current/conf/nifi.properties` in the NiFi container and ensure `nifi.web.proxy.host` matches. -

- - Possible error when dealing with non-pgsql databases `due to Incorrect syntax near 'LIMIT'.; routing to failure: com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near 'LIMIT'`, go to the GenerateTableFetch Process -> right-click -> configure -> change database type from Generic to -> MS SQL 2012 + or 2008 (if an older DB system is used) - - Possible error on Linux systems related to `nifi.properties` permission error and/or other files from the `nifi/conf/` folder, please see the [nifi doc](../nifi/main.md#important-note-about-nifi-properties) {nifi.properties} section. -

- - `Driver class org.postgresql.Driver is not found` or something similar for other MSSQL/SQL drivers, this is a known issue after NiFi version v1.20+, first, make sure you pull the latest version of the repository, then for the JAR file you are using, please execute the following command in order to verify its integrity `jar -tvf ./nifi/drivers/your_file_version.jar`, if this returns a list of files and NO errors then the files are not corrupted and can be loaded. On the NiFi side make sure to go to the `DBCPConnectionPool` controller service and verify the propertiesit a few times, make sure the file path is correct and in the following format: `file:///opt/nifi/drivers/postgresql-42.7.7.jar` for example. If all this fails stop nifi, delete all the Docker volumes associated with it -> restart NiFi, perform the above steps again. You can try forcefully starting the `GenerateTableFetch` or `QueryDatabaseTable` processors by enabling the `DBCPConnectionPool` even if an error popus up after clicking the verify button. -

- - `502 Bad Gateway`, NiFi simply not starting, even after waiting more than 2-3 minutes. This can occur due to a wide variety of issues, you can check the NiFi container log : β€œdocker logs -f --tail 1000 cogstack-nifi > my_log_file.txt” to capture the output easily. The most common cause is running out of memory, increase or decrease the limits in `nifi/conf/bootstrap.conf` according to your machine's spec, please read [bootstrap.conf](../nifi/main.md#bootstrapconf) -

- - `Unable to connect to ElasticSearch` using the `ElasticSearchClientService` NiFi controller, make sure the settings are correct (username,password,certificates, etc.) and then click `Apply`, disregard the errors and click `Enable` on the controller to forcefully reload the controller, stop it and then validate the settings, start it again after and it should work. +If NiFi reports that it supports only one mode of HTTP or HTTPS operation: -### πŸ›’οΈ **Elasticsearch Errors** +1. Check local changes in `nifi/conf/nifi.properties`. +2. Confirm the HTTPS settings loaded from `deploy/nifi.env`. +3. Recreate the NiFi containers without deleting their volumes. +4. Delete NiFi state volumes only if you have confirmed that stale persisted + configuration is the cause and the stored state is disposable. -#### ⚑ **VM memory errors, failed bootstrap check** +#### Keystore or truststore errors -It is quite a common issue for both opensearch and native-ES to error out when it comes to virtual memory allocation, this error typically comes in the form of : +Run the security initializer again; it validates complete certificate sets and +does not overwrite a valid existing set: ```bash -ERROR: [1] bootstrap checks failed -[1]: max virtual memory areas vm.max_map_count [65111] is too low, increase to at least [262144] +make -C deploy init-security ``` -To solve this one needs to simply execute : -
- - on Linux/Mac OS X : - `sysctl -w vm.max_map_count=262144` in terminal. - To make the same change systemwide plase add `vm.max_map_count=262144` to /etc/sysctl.conf and restart the dockerservice/machine. - An example of this can be found under /services/elasticsearch/sysctl.conf -
- - on Windows you need to enter the following commands in a powershell instance: -
- `wsl -d docker-desktop` -
- `sysctl -w vm.max_map_count=262144` +If it reports an incomplete certificate set, preserve the existing files and +repair or explicitly replace the set. Check the +[certificate guide](../security/certificates.md) before replacing a CA or leaf +certificate. -For more on this issue please read: https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html +#### HTTP 421 or invalid host header -
+NiFi returns `HTTP 421 INVALID PROXY` or `System Error: Invalid host header` +when it does not trust the public host and port forwarded by nginx. -#### πŸ“„ **OpenSearch: validating opensearch.yml hosts** +Set `NIFI_WEB_PROXY_HOST` in `deploy/nifi.env` to the browser-visible host and +port. For example: + +```text +NIFI_WEB_PROXY_HOST="localhost:8443,nifi.example.org:8443" +``` + +Use port `443` for `https://nifi.example.org/nifi`, or `8443` for +`https://nifi.example.org:8443/nifi`. Restart `nifi` and `nifi-nginx` after the +change. + +#### Database processor errors + +- For Microsoft SQL Server errors involving `LIMIT`, configure + `GenerateTableFetch` with the appropriate Microsoft SQL Server database type + instead of `Generic`. +- If a JDBC driver class is not found, verify the JAR before configuring its + absolute container path: + + ```bash + jar -tvf ./nifi/drivers/postgresql-42.7.7.jar + ``` + + A PostgreSQL driver path in NiFi should look like + `file:///opt/nifi/drivers/postgresql-42.7.7.jar`. Apply the controller-service + configuration, then disable and re-enable the service to reload it. + +#### 502 Bad Gateway + +A 502 from nginx usually means that NiFi is not ready or has stopped. Inspect +the container state and logs: ```bash -FATAL Error: [config validation of [opensearch].hosts]: types that failed validation: -- [config validation of [opensearch].hosts.0]: expected URI with scheme [http|https]. -- [config validation of [opensearch].hosts.1]: could not parse array value from json input +docker ps -a --filter name=cogstack-nifi +docker logs --tail 1000 cogstack-nifi ``` -This issue may appear after the recent switch to using fully customizable environment variables. Strings and ENV vars may be parsed differently depending on the shell version found on the host system. +NiFi startup can take several minutes. If the process was killed because of +memory pressure, review the Docker resource limits in `deploy/nifi.env` and the +JVM settings documented under +[bootstrap.conf](../nifi/main.md#bootstrapconf). + +### πŸ›’οΈ Elasticsearch/OpenSearch + +#### `vm.max_map_count` bootstrap failure -To solve this, the easiest way is to make sure to load the `elasticsearch.env` variables before starting the Elastic & Kibana containers by doing the following: +Search nodes may fail with an error similar to: + +```text +bootstrap checks failed +max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] +``` + +On Linux, set the value temporarily with: ```bash - cd ./deploy/ - set -a - source elasticsearch.env - make start-elastic +sudo sysctl -w vm.max_map_count=262144 ``` -Alternatively (if the script executes without issues): +For a persistent Linux configuration, add `vm.max_map_count=262144` to +`/etc/sysctl.conf` or an appropriate file under `/etc/sysctl.d/`, then apply the +configuration. On Docker Desktop, set the value in the Linux VM used by Docker. + +#### OpenSearch host configuration fails validation + +Errors saying that `opensearch.hosts` expects a URI usually indicate that the +environment was not loaded or that the value is not valid JSON/URI syntax. +Prefer the Make target, which loads the environment automatically: ```bash - cd ./deploy/ - source export_env_vars.sh - make start-elastic +make -C deploy start-elastic ``` -### πŸ—ƒοΈ DB-samples issues +If you invoke Compose directly, load the deployment environment first: ```bash -No table data for samples_db +cd deploy +source ./export_env_vars.sh +docker compose -f services.yml up -d elasticsearch-1 elasticsearch-2 kibana ``` -It is possible that you may have forgotten to pull the large files from the repo, please do : `git lfs pull`. +### πŸ—ƒοΈ Sample database has no data + +If `samples_db` contains no sample tables, make sure Git LFS assets were pulled: + +```bash +git lfs pull +``` -Delete the samples-db container and it's volumes and restart it, you should now see the data in the tables. +Then recreate the sample database only if its existing data is disposable. Stop +the service, inspect its volume with `docker volume ls`, and remove only that +deployment's sample database volume before starting `samples-db` again. diff --git a/docs/deploy/workflows.md b/docs/deploy/workflows.md index 7cae46b22..c9ec32c67 100644 --- a/docs/deploy/workflows.md +++ b/docs/deploy/workflows.md @@ -1,15 +1,71 @@ -# Workflows +# πŸ”€ Workflows -This page contains current workflow guidance for NiFi 2.x based deployments. -Historical walkthroughs and older template notes are kept separately in -[legacy workflows](workflows_legacy.md). +This section documents every maintained NiFi 2.x flow definition in +`nifi/user_templates/`. Files below `nifi/user_templates/legacy/` are historical and +are not covered. -Template locations: +## πŸ“š Maintained flow definitions -- `nifi/user_templates/` for current templates (JSON). -- `nifi/user_templates/legacy/` for older templates (XML/reference). +| Flow definition | Source | Destination | Additional service | +|---|---|---|---| +| [Database documents to OpenSearch](workflows/database_to_opensearch.md) | PostgreSQL table | OpenSearch index | None | +| [Database documents via OCR](workflows/database_ocr_to_opensearch.md) | PostgreSQL binary-document table | OpenSearch index | OCR service | +| [MedCAT annotations to OpenSearch](workflows/medcat_annotations_to_opensearch.md) | Existing OpenSearch index | Annotation index | MedCAT service | +| [Filesystem Parquet to OpenSearch](workflows/filesystem_parquet_to_opensearch.md) | Parquet files under `/data` | Index per filename | None | +| [Azure Blob Parquet to OpenSearch](workflows/azure_blob_parquet_to_opensearch.md) | Azure Blob Storage | Index per blob prefix | Azure Storage | -## Cerner blob decompression +## πŸš€ Prepare and import a flow + +From the repository root, initialize certificates and start the core data services: + +```bash +make -C deploy init-security +make -C deploy start-data-infra +``` + +Then open NiFi at `https://localhost:8443`. Keep the imported process group stopped +while configuring it. Use NiFi's **Upload Process Group** or flow-definition import +action and select the JSON file named by the relevant guide. + +Most bundled standard processors and controller services target NiFi 2.10.0; the Azure +Blob flow contains Azure components from NiFi 2.7.2. The definitions include processor +and controller-service settings, but they do not contain parameter contexts and NiFi +omits sensitive property values from exports. Review component compatibility when +importing them into another NiFi version. + +## πŸ” Post-import configuration + +Before starting an imported flow: + +1. Open the process group's **Controller Services** configuration. +2. Enter all required sensitive values. Depending on the flow, these include database, + OpenSearch, keystore, truststore, or Azure Storage credentials. +3. Confirm that service URLs, database tables, field names, certificate paths, and + target index names match the deployment. +4. Enable the record readers/writers, SSL context, database pool, storage credentials, + and OpenSearch client used by the flow. +5. Resolve every validation error, then start the process group. + +The bundled URLs and credentials are local-development defaults. Use deployment-specific +values in production. Failure, retry, and error relationships terminate at funnels in +these examples; inspect their queues instead of assuming failed FlowFiles were discarded. + +## πŸ”Ž Verify OpenSearch output + +Check the expected index in OpenSearch Dashboards or query its count endpoint. For the +default local OpenSearch deployment: + +```bash +curl --cacert security/certificates/elastic/opensearch/elastic-stack-ca.crt.pem \ + -u ':' \ + 'https://localhost:9200//_count' +``` + +Use the target index documented for the selected flow. A successful HTTP response does +not prove that every FlowFile succeeded, so also inspect NiFi bulletins, provenance, and +the queues connected to failure funnels. + +## πŸ—œοΈ Cerner blob decompression Use `CogStackJsonRecordDecompressCernerBlob` when source documents are stored as Cerner blob fragments, with one database row per blob sequence. The processor expects one @@ -17,7 +73,7 @@ FlowFile to contain all blob rows for a single document, then sorts the fragment sequence number, validates the sequence, concatenates the binary payload, and extracts or decompresses the embedded document bytes. -Recommended processor chain: +### πŸ”— Recommended processor chain ```text Fetch document IDs @@ -32,7 +88,7 @@ that every blob sequence for a document remains in the same FlowFile. For Cerner rows, prefer a document-ID driven query where each `ExecuteSQLRecord` invocation fetches all sequences for one document. -Example SQL shape: +### 🧾 Example SQL shape ```sql SELECT @@ -44,7 +100,7 @@ WHERE docid = ? ORDER BY blob_sequence_num ``` -`ExecuteSQLRecord` settings: +### βš™οΈ `ExecuteSQLRecord` settings - Set `Max Rows Per FlowFile` to `0`. - Set `Output Batch Size` to `0`. @@ -56,6 +112,8 @@ ORDER BY blob_sequence_num SQL is configured on the processor, pass the document ID as a prepared-statement argument, for example with `sql.args.1.type` and `sql.args.1.value` attributes. +### πŸ“₯ Required input fields + Input records must include these fields unless you override the processor properties: | Field | Default property | Required behavior | @@ -64,7 +122,7 @@ Input records must include these fields unless you override the processor proper | Blob sequence | `blob_sequence_order_field_name=blob_sequence_num` | Present, integer-like, unique by default, contiguous, and starting at `0` or `1`. | | Blob payload | `binary_field_name=binarydoc` | Present and non-empty. With the default `binary_field_source_encoding=base64`, this must be a base64 string. | -Recommended processor properties: +### πŸ› οΈ Recommended processor properties | Property | Recommended value | Notes | |---|---|---| @@ -75,6 +133,8 @@ Recommended processor properties: | `binary_field_source_encoding` | `base64` | Recommended for JSON FlowFiles. | | `output_mode` | `base64` | Recommended for JSON output and downstream processors. | +### ⚠️ Failure routing + The processor routes to `failure` when it detects unsafe input, including: - missing document ID, blob sequence, or blob payload fields; @@ -84,6 +144,8 @@ The processor routes to `failure` when it detects unsafe input, including: - invalid base64 payloads; - payloads that cannot be extracted as embedded PDF/RTF bytes or decoded as Cerner LZW. +### βœ… Successful output + On success, the output is a JSON array containing one merged record. Non-binary fields are copied from the first input record, and the blob field contains the decompressed or extracted document bytes encoded according to `output_mode`. Useful FlowFile attributes diff --git a/docs/deploy/workflows/azure_blob_parquet_to_opensearch.md b/docs/deploy/workflows/azure_blob_parquet_to_opensearch.md new file mode 100644 index 000000000..56c2e9c85 --- /dev/null +++ b/docs/deploy/workflows/azure_blob_parquet_to_opensearch.md @@ -0,0 +1,90 @@ +# ☁️ Azure Blob Parquet to OpenSearch + +Template: `nifi/user_templates/azure_blobs_parquet_to_opensearch.json` + +[View the flow definition on GitHub](https://github.com/CogStack/CogStack-NiFi/blob/main/nifi/user_templates/azure_blobs_parquet_to_opensearch.json) + +## 🎯 Purpose + +This flow lists blobs in an Azure Storage container, fetches Parquet blobs, converts +them to NDJSON, and writes their records to OpenSearch. The first path segment of each +blob name becomes the destination index. + +## πŸ”— Processor chain + +```text +ListAzureBlobStorage_v12 + -> FetchAzureBlobStorage_v12 + -> RouteOnAttribute-parquetFiles + -> ExecuteStreamCommand-ParquetToJson + -> PutElasticsearchRecord +``` + +The list processor emits blob metadata, the fetch processor retrieves one blob per +FlowFile, and `RouteOnAttribute` accepts filenames ending in `.parquet`. The conversion +script reads the Parquet content from standard input and emits NDJSON to standard output. + +## πŸ“‹ Bundled defaults + +| Setting | Value | +|---|---| +| Azure endpoint suffix | `blob.core.windows.net` | +| Credential mode | Account key | +| Container name | `container_name` placeholder | +| Listing strategy | Timestamps | +| Polling schedule | `1 min` | +| Initial listing target | All blobs | +| Accepted filename suffix | `.parquet` | +| Conversion command | `python3.11 convert_record_parquet_to_json.py` | +| Script working directory | `/opt/nifi/user_scripts/processors/` | +| Target index | `${azure.blobname:substringBefore('/'):toLower()}` | +| Index operation | `create` | +| OpenSearch bulk batch size | `10000` | + +The index expression expects blob names shaped like `/.parquet`. Review it +before processing root-level blobs or prefixes containing characters that are invalid in +OpenSearch index names. + +## βœ… Requirements + +- NiFi and OpenSearch are running and certificates are generated. +- The NiFi container can reach the Azure Storage endpoint. +- The Azure account, container, and selected authentication method allow blob listing + and reads. +- The bundled Parquet conversion script and Python 3.11 are available in the NiFi image. +- Azure, OpenSearch, keystore, and truststore sensitive values are entered after import. + +## βš™οΈ Configure after import + +1. In `AzureStorageCredentialsControllerService_v12-RIO_BLOBS`, enter the storage + account name and account key, or select and configure another supported credential + type. Do not place production secrets in the flow JSON. +2. Set the real container name on `ListAzureBlobStorage_v12`. +3. Keep its **Record Writer** unset. The flow expects one listed blob per FlowFile so + `FetchAzureBlobStorage_v12` can use `${azure.container}` and `${azure.blobname}`. +4. Confirm the route expression and conversion command path. +5. Configure and enable the JSON reader/writer, SSL context, Azure credential service, + and OpenSearch client. +6. Review the target-index expression and add a deterministic `ID Record Path` if the + Parquet data has a stable unique key. + +## ▢️ Run and verify + +Start with a dedicated test prefix and a small Parquet blob. Verify the derived index, +then compare its count with the source file: + +```bash +curl --cacert security/certificates/elastic/opensearch/elastic-stack-ca.crt.pem \ + -u ':' \ + 'https://localhost:9200//_count' +``` + +## ⚠️ State and failure handling + +The list processor keeps timestamp-based state and is configured to include all blobs +on its initial listing. Test the scope before enabling it against a large container. +Resetting listing state can fetch old blobs again. Because the writer uses `create` +without a document ID, reprocessing can create duplicate logical records. + +Fetch failures, conversion non-zero exits, OpenSearch failures, retries, and per-record +error responses are routed to funnel queues for inspection. diff --git a/docs/deploy/workflows/database_ocr_to_opensearch.md b/docs/deploy/workflows/database_ocr_to_opensearch.md new file mode 100644 index 000000000..17d1d15a0 --- /dev/null +++ b/docs/deploy/workflows/database_ocr_to_opensearch.md @@ -0,0 +1,108 @@ +# πŸ“„ Database documents via OCR + +Template: `nifi/user_templates/opensearch_ingest_docs_db_ocr_service_to_es.json` + +[View the flow definition on GitHub](https://github.com/CogStack/CogStack-NiFi/blob/main/nifi/user_templates/opensearch_ingest_docs_db_ocr_service_to_es.json) + +## 🎯 Purpose + +This flow reads binary documents from PostgreSQL, sends one document at a time to the +CogStack OCR service, merges the extracted text with the source metadata, and indexes +the result in OpenSearch. + +## πŸ”— Processor chain + +```text +GenerateTableFetch-encounters + -> ExecuteSQLRecord + -> CogStackPrepareRecordForOcr + -> InvokeHTTP + -> CogStackParseCogStackServiceResult + -> SplitJson + -> EvaluateJsonPath + -> PutElasticsearchJson +``` + +`CogStackPrepareRecordForOcr` converts the Avro binary field to a base64 JSON request +and preserves the remaining source fields in a footer. After OCR, the parser restores +that footer, adds the extracted `text`, and exposes `id` as a FlowFile attribute for +OpenSearch indexing. + +## πŸ“‹ Bundled defaults + +| Setting | Value | +|---|---| +| Database URL | `jdbc:postgresql://samples-db:5432/db_samples` | +| Source table | `encounters` | +| Incremental/partition column | `cid` | +| Partition size | `1000` | +| Rows per FlowFile | `1` | +| Binary source field | `binarydocument` | +| Document ID field | `id` | +| OCR endpoint | `http://ocr-service:8090/api/process` | +| Extracted text field | `text` | +| OpenSearch endpoint | `https://elasticsearch-1:9200` | +| Target index | `${generatetablefetch.tableName}` (`encounters` by default) | +| Index operation | `index` | +| OpenSearch identifier attribute | `id` | + +## βœ… Requirements + +- The core stack and sample database are running. +- The OCR service is running. From the repository root use: + + ```bash + make -C deploy start-ocr-services + ``` + +- The source table contains a numeric incremental column, a unique document ID, and a + binary document column. +- NiFi can resolve `ocr-service` on the Docker network, or the `InvokeHTTP` URL is + changed to a reachable endpoint. +- Database, OpenSearch, keystore, and truststore passwords are entered after import. + +## βš™οΈ Configure after import + +1. Configure and enable `DBCPConnectionPool` for the source database. +2. Update `GenerateTableFetch-encounters` when the source table or incremental column + differs from `encounters.cid`. +3. Set `CogStackPrepareRecordForOcr` properties to the actual source fields: + + | Property | Bundled value | + |---|---| + | `process_flow_file_type` | `avro` | + | `binary_field_name` | `binarydocument` | + | `document_id_field_name` | `id` | + | `output_text_field_name` | `text` | + | `operation_mode` | `base64` | + +4. Verify the `InvokeHTTP` URL. For the text-extraction-only container on the same + Docker network, use `http://ocr-service-text-only:8090/api/process`. +5. Keep **Request Body Enabled** set to `true`; the processor sends JSON rather than a + raw binary body. +6. Configure the SSL context and OpenSearch client passwords, hosts, and certificate + paths. Verify that `EvaluateJsonPath` extracts the same ID field used by + `PutElasticsearchJson`. + +## ▢️ Run and verify + +Start the process group with a small source range first. Confirm that OCR responses +reach `CogStackParseCogStackServiceResult`, that split records have an `id` attribute, +and that the destination index count increases: + +```bash +curl --cacert security/certificates/elastic/opensearch/elastic-stack-ca.crt.pem \ + -u ':' \ + 'https://localhost:9200/encounters/_count' +``` + +The default flow writes OCR-enriched records to an index with the same name as the +source table. Change the target index before starting if source and enriched documents +must be kept separately. + +## ⚠️ Failure handling + +HTTP failures, parse failures, SQL failures, unmatched IDs, and OpenSearch errors are +routed to separate funnel queues. Inspect the response body and NiFi bulletin before +retrying. Clearing `GenerateTableFetch` state causes source rows to be submitted to OCR +again and should be done only deliberately. diff --git a/docs/deploy/workflows/database_to_opensearch.md b/docs/deploy/workflows/database_to_opensearch.md new file mode 100644 index 000000000..3c6b611f6 --- /dev/null +++ b/docs/deploy/workflows/database_to_opensearch.md @@ -0,0 +1,92 @@ +# πŸ—„οΈ Database documents to OpenSearch + +Template: `nifi/user_templates/opensearch_ingest_docs_db_to_es.json` + +[View the flow definition on GitHub](https://github.com/CogStack/CogStack-NiFi/blob/main/nifi/user_templates/opensearch_ingest_docs_db_to_es.json) + +## 🎯 Purpose + +This flow incrementally reads text documents from a PostgreSQL table and indexes each +record in OpenSearch. The bundled configuration targets the sample +`medical_reports_text` table and uses `docid` as the stable document identifier. + +## πŸ”— Processor chain + +```text +GenerateTableFetch-medical_reports_text + -> ExecuteSQLRecord + -> PutElasticsearchRecord +``` + +- `GenerateTableFetch` creates paginated SQL queries and tracks the maximum `docid` it + has seen. +- `ExecuteSQLRecord` executes each query and writes embedded-schema Avro. +- `PutElasticsearchRecord` reads the Avro records and indexes them using `/docid` as + the OpenSearch document ID. + +Failure and retry relationships are routed to funnels for inspection. + +## πŸ“‹ Bundled defaults + +| Setting | Value | +|---|---| +| Database URL | `jdbc:postgresql://samples-db:5432/db_samples` | +| Database user | `test` | +| JDBC driver | `/opt/nifi/drivers/postgresql-42.7.7.jar` | +| Source table | `medical_reports_text` | +| Incremental/partition column | `docid` | +| Partition size | `10000` | +| Returned columns | `sampleid`, `typeid`, `dct`, `filename`, `document`, and `docid` cast to text | +| OpenSearch endpoint | `https://elasticsearch-1:9200` | +| Target index | `${generatetablefetch.tableName}` (`medical_reports_text` by default) | +| Index operation | `index` | +| Document ID | `/docid` | +| Bulk batch size | `10000` | + +## βœ… Requirements + +- The core stack is running; for the bundled sample database use + `make -C deploy start-data-infra`. +- The source table has a numeric, indexed, monotonically increasing maximum-value + column. The sample schema provides integer primary key `docid`. +- OpenSearch certificates exist and are mounted into NiFi. +- Database and OpenSearch credentials are available. Sensitive passwords are not + included in the exported JSON. + +## βš™οΈ Configure after import + +1. In `DBCPConnectionPool`, verify the JDBC URL, driver class, driver location, user, + and password. The bundled local values correspond to `security/env/users_database.env`. +2. In `GenerateTableFetch`, change the table name, returned columns, partition column, + and maximum-value column when using another schema. +3. In `PutElasticsearchRecord`, verify the target index and set `ID Record Path` to the + unique source field, including the leading `/` required by RecordPath. +4. In `StandardSSLContextService`, enter the keystore, key, and truststore passwords and + confirm that the mounted files match the selected search backend. +5. In `ElasticSearchClientServiceImpl`, set the correct hosts and credentials, then + enable every controller service. + +`Use Avro Logical Types` is enabled. If a target mapping rejects dates or timestamps, +adjust the record conversion and OpenSearch mapping deliberately rather than disabling +logical types without checking the resulting JSON. + +## ▢️ Run and verify + +Start the process group and watch the queues between all three processors. The default +target is `medical_reports_text`: + +```bash +curl --cacert security/certificates/elastic/opensearch/elastic-stack-ca.crt.pem \ + -u ':' \ + 'https://localhost:9200/medical_reports_text/_count' +``` + +Because the flow uses `docid` and the `index` operation, processing the same source row +again replaces the document with that ID instead of creating another generated ID. + +## ⚠️ State and recovery + +`GenerateTableFetch` stores incremental state. Clear its state only when intentionally +re-reading older rows. Before doing so, confirm that the target document ID is stable and +that re-indexing is safe. Check the failure funnels and NiFi bulletins before advancing +or discarding queued FlowFiles. diff --git a/docs/deploy/workflows/filesystem_parquet_to_opensearch.md b/docs/deploy/workflows/filesystem_parquet_to_opensearch.md new file mode 100644 index 000000000..0e84f254b --- /dev/null +++ b/docs/deploy/workflows/filesystem_parquet_to_opensearch.md @@ -0,0 +1,86 @@ +# πŸ“ Filesystem Parquet to OpenSearch + +Template: `nifi/user_templates/opensearch_ingest_parquet_form_fs_to_es.json` + +[View the flow definition on GitHub](https://github.com/CogStack/CogStack-NiFi/blob/main/nifi/user_templates/opensearch_ingest_parquet_form_fs_to_es.json) + +## 🎯 Purpose + +This flow recursively discovers Parquet files in NiFi's `/data` directory, converts +each file to newline-delimited JSON (NDJSON), and writes its records to an OpenSearch +index derived from the filename. + +## πŸ”— Processor chain + +```text +GetFile + -> CogStackConvertParquetToJson + -> PutElasticsearchRecord +``` + +The custom Python processor reads Parquet in batches of 10,000 records and emits one +compact JSON object per line. It adds `mime.type=application/x-ndjson` and a +`record.count` FlowFile attribute. + +## πŸ“‹ Bundled defaults + +| Setting | Value | +|---|---| +| NiFi input directory | `/data/` | +| Host directory | `data/` by default through the Compose mount | +| Recursive scan | Enabled | +| File filter | `.*\.parquet` | +| Polling interval | `30 sec` | +| Files per poll | `10` | +| Keep source file | `true` | +| Target index | `${filename:substringBefore('.parquet'):toLower()}` | +| Index operation | `create` | +| OpenSearch bulk batch size | `10000` | + +## βœ… Requirements + +- NiFi and OpenSearch are running and their generated certificates are available. +- The NiFi Python environment includes `pyarrow`; the repository image installs the + extension dependencies. +- Parquet files are readable by the NiFi container below `/data`. +- OpenSearch and SSL controller-service passwords are entered after import. + +The standard deployment mounts `${NIFI_DATA_PATH:-../data/}` at `/data/`. With default +settings, place input files under the repository's `data/` directory. + +## βš™οΈ Configure after import + +1. Confirm the `GetFile` input directory and file filter. Use a mounted path; a host path + that is not visible inside the container will not work. +2. Set **Keep Source File** to `false` for one-time ingestion and ensure NiFi can delete + the source file. When it is `true`, `GetFile` picks up the same file continually; use + a `ListFile`/`FetchFile` pattern or another archival strategy if files must remain. +3. Configure the SSL context and `ElasticSearchClientServiceImpl` credentials and + certificate passwords. +4. Review the index expression. `example.parquet` becomes index `example`. Files with + the same name in different subdirectories target the same index. +5. Add an `ID Record Path` or another deterministic ID strategy when the source contains + a stable key and safe reprocessing is required. + +## ▢️ Run and verify + +Copy a small Parquet file into `data/`, start the process group, and verify that +`record.count` matches the number of indexed rows: + +```bash +curl --cacert security/certificates/elastic/opensearch/elastic-stack-ca.crt.pem \ + -u ':' \ + 'https://localhost:9200//_count' +``` + +## ⚠️ Reprocessing and failures + +The bundled **Keep Source File** value is `true`, so `GetFile` continually reprocesses +the same files. The writer also uses `create` without a configured document ID, causing +OpenSearch to generate new IDs and duplicate logical records. Change the retention +behavior before starting the flow unless repeated test ingestion is intentional. + +See the +[Apache NiFi `GetFile` documentation](https://nifi.apache.org/components/org.apache.nifi.processors.standard.GetFile/) +for the processor's source-file behavior. Conversion and OpenSearch failures are routed +to funnels; the writer also exposes detailed `error_responses` for rejected records. diff --git a/docs/deploy/workflows/medcat_annotations_to_opensearch.md b/docs/deploy/workflows/medcat_annotations_to_opensearch.md new file mode 100644 index 000000000..8bdf578e2 --- /dev/null +++ b/docs/deploy/workflows/medcat_annotations_to_opensearch.md @@ -0,0 +1,97 @@ +# 🧠 MedCAT annotations to OpenSearch + +Template: `nifi/user_templates/opensearch_ingest_docs_annotations_medcat_service_to_es.json` + +[View the flow definition on GitHub](https://github.com/CogStack/CogStack-NiFi/blob/main/nifi/user_templates/opensearch_ingest_docs_annotations_medcat_service_to_es.json) + +## 🎯 Purpose + +This flow reads documents that already exist in OpenSearch, sends their text to the +CogStack MedCAT service, expands the response into individual annotation documents, +and writes those annotations to a separate OpenSearch index. + +## πŸ”— Processor chain + +```text +SearchElasticsearch + -> CogStackPrepareRecordForNlp + -> InvokeHTTP + -> CogStackParseCogStackServiceResult + -> SplitJson + -> EvaluateJsonPath-create_annotation_id_store_as_attritbute + -> PutElasticsearchJson +``` + +The preparation processor sends the document text plus a footer containing the source +fields. The response parser creates an `annotation_id` by combining the source document +ID and MedCAT annotation ID. `EvaluateJsonPath` copies it to a FlowFile attribute used +as the destination document ID. + +## πŸ“‹ Bundled defaults + +| Setting | Value | +|---|---| +| Source OpenSearch index | `encounters` | +| Source fields | `id`, `text` | +| Search condition | Documents where `text` exists | +| Pagination | Scroll, keep-alive `30 mins` | +| Restart on finish | `false` | +| MedCAT endpoint | `http://cogstack-medcat-service-production:5000/api/process_bulk` | +| Document ID field | `id` | +| Document text field | `text` | +| Destination index | `encounters_annotations` | +| Index operation | `index` | +| Destination ID attribute | `annotation_id` | + +## βœ… Requirements + +- The source OpenSearch index exists and its documents contain stable `id` and `text` + fields. +- The MedCAT service and its configured model are running: + + ```bash + make -C deploy start-medcat-service + ``` + +- NiFi can resolve `cogstack-medcat-service-production` on the Docker network, or the + `InvokeHTTP` URL is changed to a reachable MedCAT endpoint. +- OpenSearch and SSL controller-service passwords are entered after import. + +## βš™οΈ Configure after import + +1. Configure `ElasticSearchClientServiceImpl` with the correct hosts and credentials. + The bundled flow lists `elasticsearch-1` and `elasticsearch-2`. +2. Configure and enable `StandardSSLContextService` using the generated deployment + certificates and passwords. +3. Update the source index, `_source` fields, and query in `SearchElasticsearch`. +4. Match `document_id_field_name` and `document_text_field_name` in + `CogStackPrepareRecordForNlp` and `CogStackParseCogStackServiceResult` to the source + mapping. +5. Verify the MedCAT URL and keep **Request Body Enabled** set to `true`. +6. Set the destination index in `PutElasticsearchJson`. Keep `annotation_id` as the + identifier unless another deterministic uniqueness scheme is required. + +The bundled parser uses normal annotation mode (`medcat_output_mode=not_set`), not +de-identification mode. Each annotation becomes a separate destination document. + +## ▢️ Run and verify + +The search processor is configured for a single pass because **Restart On Finish** is +`false`. Start with a restrictive query or a test index, then confirm the annotation +index count: + +```bash +curl --cacert security/certificates/elastic/opensearch/elastic-stack-ca.crt.pem \ + -u ':' \ + 'https://localhost:9200/encounters_annotations/_count' +``` + +Using deterministic `annotation_id` values makes deliberate reprocessing idempotent for +the same source document and MedCAT annotation IDs. + +## ⚠️ Failure handling + +Search, HTTP, preparation, parsing, splitting, identifier extraction, and OpenSearch +failures are routed to funnels. Documents with no returned annotations do not create +annotation records in the bundled non-de-identification mode. Inspect MedCAT responses +and NiFi provenance when input counts and annotation counts differ. diff --git a/docs/deploy/workflows_legacy.md b/docs/deploy/workflows_legacy.md deleted file mode 100644 index 5ec87d424..000000000 --- a/docs/deploy/workflows_legacy.md +++ /dev/null @@ -1,428 +0,0 @@ -# πŸ”€ Workflows - -## Current status (NiFi 2.x) - -This page contains legacy workflow walkthroughs that are still useful for reference, but some processor names, properties, and template formats have changed in NiFi 2.x. - -For current and maintained guidance, use: - -- [Current workflows](./workflows.md) -- [Prerequisites and deployment overview](./main.md) -- [Deployment operations guide](./deployment.md) -- [Services reference](./services.md) -- [NiFi guide](../nifi/main.md) -- [Processor scripting guide](../nifi/processor_scripting.md) - -Template locations: - -- `nifi/user_templates/` for current templates (JSON). -- `nifi/user_templates/legacy/` for older templates (XML/reference). - -## Legacy workflow examples (reference) - -!!! warning - - These examples are maintained as historical reference material. Validate processor configuration and controller-service properties against your currently deployed NiFi version. - -Our custom Apache NiFi image comes with 4 basic example template workflows bundled that available in [user templates](https://github.com/CogStack/CogStack-NiFi/tree/main/nifi/user_templates) in `./nifi` directory. -These are: - -1. `OpenSearch_ingest_DB_to_ES` - performing ingestion of free-text notes from database to Elasticsearch, no pre-processing involved. -2. `OpenSearch_ingest_DB_OCR_service_to_ES` - performing ingestion of raw notes in PDF format from database to Elasticsearch, OCR involved using the OCR-service. -3. `OpenSearch_ingest_annotate_DB_MedCATService_to_ES` - annotating the free-text notes using MedCAT, reading from database and storing in Elasticsearch. -4. `OpenSearch_ingest_annotate_ES_MedCATService_to_ES` - the same as (3) but reading notes from Elasticsearch. - -If you are using Nifi with SSL mode (which is on by default as of the upgrade to version 1.15+), please note that all of these templates have SSL configured (SSLContext service controller being present), please make sure that you set the password(s) to the key/trust(store) for the templates to work. - -There are more workflows available in the sections below - - IMPORTANT: if you do not see some workflows in the NiFi Template Web interface then you will have to manually go to the `./nifi/user_templates` folder and upload whatever templates are missing, the reason for this is that NiFi keep its own available template(s) file separately and we do not update this as it will overwrite the user's own file. - -
- -## Used services -In the workflow examples, the following services are used: -- `samples-db` - storing the example input data, -- `nifi` - the actual Apache NiFi data pipeline engine with user interface, -- `elasticsearch-1` - for storing the resulting documents and annotations data, cluster of two instances. -- `elasticsearch-2` - second node -- `tika-service` - extraction of text from binary documents, -- `nlp-medcat-service-production` - an example NLP application for extracting annotations from free-text. -- `cogstack-cohort` - CogStack-Cohort tool -- `ocr-service-1`/`ocr-service-2` - OCR service(s) - -To deploy the above services, one can type in the `deploy` directory: -``` -make init-security -make start-data-infra -make start-medcat-service -``` - -Please note that all the above services will be accessible by services within internal `cognet` Docker network while only some of them will be accessible from host machine. -Please refer to [SERVICES](./services.md) for a more detailed description of the available services and their deployment. - -
- -## Apache NiFi web user interface -Before start, please see [the official Apache NiFi guide on using the web user interface](https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#User_Interface) that covers extensively the available functionality. - -In this doc only the key aspects will be covered on using the bundled user templates with configuring and executing the flows. - -Once deployed, the Apache-NiFi web interface will be accessible from the host (e.g. localhost) machine at `https://localhost:8443`. - -To see all available user workflow templates navigate to **Templates** window by clicking the corresponding list item as presented on the figure below. -Following, to select an example workflow template to run, drag and drop the **template** button from the components toolbar panel to the main notepad window. -![template-w1](../_static/img/legacy_images/nifi-templates-w1.png) - - -Please note that all the available workflow templates that are bundled with our custom Apache NiFi image are available in [`../nifi/user_templates`](https://github.com/CogStack/CogStack-NiFi/tree/main/nifi/user_templates) directory. -During normal work, the user has possibility to create and store own template workflows. -These workflows are represented as XML files and so can be easily further shared or modified. - -The templates mentioned in the introduction section may REQUIRE some minor configuration such as adding the passwords to the ES connectors (u: `admin`, pw:`admin`) and to the SSLContext Service Controller, -the trust/key(store) password is `cogstackNifi`. - -Every time you make change to the SSL Service Controller you MUST verify/validate all the properties first, - -
- -## Ingesting free-text documents (DB β†’ ES) -This workflow implements a common data ingestion pipeline: reading from a database and storing the free-text data alongside selected structured fields into Elasticsearch. -The workflow was presented on the figure above. - -
- -### Reading records from database -Free-text data alongside available structured fields are read from `samples-db` database from `medical_reports_text` table. -This operation is implemented by NiFi components: `GenerateTableFetch` and `ExecuteSQLRecord`, where the configuration of the former component is described on the picture below. -The `docid` field is set as the primary key of the `medical_reports_text` table and is used persist the state of the last read record and to partition the records while reading. -![db-reader](../_static/img/legacy_images/configure-db-reader-w1.png) - -
- -### Configuring DB connector -However, apart from specifying the DB tables, the DB connector controller `DBCPConnectionPool-MTSamples` needs to configured and activated. -The example data is stored in `db_samples` database. -User `test` with password `test` was created to connect to it. - -Alongside the DB connector, other controllers used by the processors (i.e. record readers and writers) need to be activated too - all of this is illustrated on the picture below. -![db-reader-w1](../_static/img/legacy_images/configure-db-connector-w1.png) - -
- -#### Parameters to lookout for - -The `DBCConnectionPool` controller service can be configured to operate with multiple DB types, you will need to configure them manually with the appropiate drivers and driver classes, the following configs can be used: -``` - PgSQL: - - Database Connection URL: jdbc:postgresql://samples-db:5432/db_samples - - Database Driver Class Name: org.postgresql.Driver - - Database Driver Location(s): /opt/nifi/drivers/postgresql-42.7.7.jar -``` -``` - MSSQL: - - Database Connection URL: jdbc:sqlserver://cogstack-databank-db-mssql:1433;DatabaseName=MTSamples;encrypt=true;trustServerCertificate=true; - - Database Driver Class Name: com.microsoft.sqlserver.jdbc.SQLServerDriver - - Database Driver Location(s): /opt/nifi/drivers/mssql-jdbc-13.2.0.jre11.jar -``` - -Reminder: the datbase driver location path is mounted on the NiFi container by default on service startup, `./nifi/drivers/:/opt/nifi/drivers/`, you may want to check the [nifi drivers folder](https://github.com/CogStack/CogStack-NiFi/tree/main/nifi/drivers) for available drivers, and if you require other drivers, please copy the .jar files there and they will be available on the NiFi container during runtime (no NiFi service restart is required) - -
- -## Adding your own data to the DB - -The easiest way to do this is to create your own sql schema file (to keep things separated) stored in the pgsamples folder( for example `services/pgsamples/new_schema.sql​` ) put your sql code there and mount it on the pg-samples container, like so: -``` - samples-db: - image: postgres:11.4-alpine - container_name: cogstack-samples-db - restart: always - volumes: - # mapping postgres data dump and initialization - - ../services/pgsamples/new_schema.sql:/data/new_schema.sql:ro # <----- this is the new line -``` -Pay attention to the mapped volume file path, we added a new line. - -Afterwards, in the `services/pgsamples/init_db.sh​ file`, you will need to add a line that imports the sql file created directly when the DB container is first initialized - -``` -# create schemas -# -echo "Defining DB schemas" -psql -v ON_ERROR_STOP=1 -U $DB_USER -d $DB_NAME -f $DATA_DIR/"new_schema.sql" -``` - -After this is done , the only things that need to be changed are in the NiFi flow config, as follows: - - - in the "GenerateTableFetch" process (make sure it is not running, and that you right click it go to view state​ and then click on the "Clear State" button, this resets the ingested records), we change the table_name from medical_reports_text to your own custom `index_table_name` , and the Maximum-value Columns to any field that we wish to select DB rows by, preferably some unique ID field that could also be a primary key `unique_id_column` ( if it's the actual key you wish to select the documents by) . - - in the "PutElasticsearchRecord" process, change the "Identifier Record Path" from "/docid" to "/unique_id_column" - -Additional steps may be required : delete current Db-samples container, then the volume (samples-vol) and restart this container, of course the - -Restart all nifi-processes and the ingestion should work. - -
- -## Indexing records by Elasticsearch -The records are finally stored in Elasticsearch data store under index `medical_reports_text` and using url endpoint `http://elasticsearch-1:9200`. -This operation is implemented by NiFi component `PutElasticsearchRecord` with its configuration presented on the picture below. -Provide the appropriate Elasticsearch/OpenSearch credentials from your deployment configuration. Local development values are defined in `security/env/users_elasticsearch.env`; do not copy them into a production flow. -When indexing the records as documents the record's primary key field `/docid` will be used as the document identifier in Elasticsearch. -Optionally, the default Date / Time / Timestamp Format can be overridden for corresponding fields being ingested. -In this example case, the Timestamp Format was overridden as `yyyy-MM-dd'T'HH:mm:ss.SSS`. -![es-writer-w1](../_static/img/legacy_images/configure-es-writer-w1.png) - -
- -## Executing the workflow -Once the NiFi components are properly configured and required connectors and controllers are activated, one can run the ingestion pipeline. -To run the pipeline, one needs to select the workflow components and click on the run button ( **β–Ί** ) on the operations panel. -Similarly, to stop execution, click on the stop button ( **β– ** ). -At any moment, one can stop and resume execution either of the full workflow or individual components to interactively inspect or troubleshoot the data processing. -The figure below presents how to execute the current workflow. -![nifi-exec-w1](../_static/img/legacy_images/nifi-exec-workflow-w1.png) - -Assuming that the services are available to be accessed on the host machine `localhost`, one can check whether the records have been indexed by Elassticsearch directly in Kibana interface by navigating to `http://localhost:5601`. - -Alternatively, one can run `curl` on local machine to check the number of documents ingested: - - ``` - curl -s -XGET http://admin:admin@localhost:9200/medical_reports_text/_count | jq - ``` - -with the expected response: - - ``` - { - "count": 259, - "_shards": { - "total": 1, - "successful": 1, - "skipped": 0, - "failed": 0 - } - } - ``` - -
- -## Ingesting text from PDF documents (DB β†’ ES) -This workflow implements an extended version of the initial data ingestion pipeline. -This time, the documents are stored in the database in binary format and so the text needs to be extracted from them prior to being indexed in Elasticsearch. -The text extraction is handled by Apache Tika that is running as Tika Service (see: [description of all available services](./services.md)). -Figure below presents the full workflow. -![nifi-w2](../_static/img/legacy_images/nifi-workflow-w2.png) - -Please note that in contrast to the previous example, this one introduces a conditional flow. -In case of a processing failure, the record (Flow File) in question will be routed via a corresponding error path. -This enables further inspecting and isolating invalid or unsupported payloads, such as when encountered by Tika a corrupted file instead of an actual document PDF file. - -
- -## Reading records from database -Similarly, as in the previous workflow, the records are read from the same database using the same database connector controller `DBCPConnectionPool-MTSamples`. -This time, the documents data are read from table `medical_reports_raw` and such columns are being queried: `docid, sampleid, dct, binarydoc`, where the `binarydoc` column contains the binary data of the documents. -However, this time we limit the database reader to fetch one row at once as configured in a single `QueryDatabaseTable` component (instead of two: `GenerateTableFetch` and `ExecuteSQLRecord`). -This is in order to have a more granular control over possibly failed documents by Tika that can be directly inspected and to avoid out-of-memory exceptions when a bulk of large scanned documents would be fetched at once. - -The figure below presents the configuration of `GenerateTableFetch` NiFi component covering the above description. -![nifi-reader-w2](../_static/img/legacy_images/configure-db-reader-w2.png) - -
- -## Extracting text from PDFs -The text extraction is implemented by Tika Service that exposes RESful API for processing the documents. -Given a binary document sent as a stream, it will return payload containing the extracted text with document metadata. - -There are 4 NiFi components involved in this process: -1. `ExecuteScript-PrepareTikaContent` - prepares the payload to be sent to Tika Service, -2. `InvokeHTTP-QueryTika` - sends the document to Tika Service and received back the response in JSON, -3. `ExecuteScript-ParseTikaResponse` - parses the Tika JSON response. -4. `ExecuteGroovyScript-ConvertJsonToAvro` - parses the JSON content into AVRO record format. - -Components (1), (3) and (4) execute custom Groovy scripts to parse the records (Flow Files). -These scripts are bundled with our custom Apache NiFi image and are available in [`./nifi/user_scripts`](https://github.com/CogStack/CogStack-NiFi/tree/main/nifi/user_scripts). - -The key component (2) is a generic HTTP client for communication with RESTful services. -It sends the binary payload using `POST` method to `http://tika-service:8090/api/process` endpoint. -Please note that same instance of Tika Service can be used by different HTTP clients in multiple data pipelines. -See [the official Tika Service documentation](https://github.com/CogStack/tika-service/) for more information about the service and API use. -Figure below shows the configuration of the HTTP client. -![configure-tika-w2](../_static/img/legacy_images/configure-tika-w2.png) - -Please note that these 4 components can be merged into a specialised component for communicating with Tika Service. - -
- -## Indexing records by Elasticsearch -This example uses the same configuration for `PutElasticsearchRecord` component as before, but the records are now stored under `medical_reports_text_tika` index. - -
- -## Annotating free-text documents (DB β†’ ES) -This workflow implements the NLP annotations ingestion pipeline based on the previous examples. -The documents are stored in the initial database in free-text format, but we are interested in extracting only the NLP annotations. -The annotations will extracted from free-text notes via NLP Service. -Finally, the annotations will be stored in Elasticsearch. - -The annotations extraction is provided by MedCAT that is exposing NLP model functionality via MedCAT Service (see: [description of all available services](./services.md)). -Figure below presents the full workflow. -![nifi-w3](../_static/img/legacy_images/nifi-workflow-w3.png) - -## Reading records from database -The documents are being read from the database by NiFi components `GenerateTableFetch` and `ExecuteSQLRecord` with the same configuration as in the first example. -However, only the `docid, document, sampleid` columns are read being the relevant ones. - -
- -## Extracting NLP annotations from documents -The annotations extraction is implemented by MedCAT Service that exposes RESful API for processing the documents. -Given a document content encoded as JSON, it will return payload containing the extracted annotations. - -There are several NiFi components involved in this process which stand out: -1. `ConvertAvroToJSON` - converts the AVRO records to JSON format using a generic format transcoder, -2. `ExecuteScript-ConvertRecordToMedCATinput` - prepares the JSON payload for MedCAT Service, this is Jython script, it has several configurable process properties: - - `document_id_field` = `docid`, the exact name of the unique Id column for the DB/ES record - - `document_text_field` = `document`, field/column name containing free text - - `log_file_name` = `nlp_request_bulk_parse_medical_text.log`, creates a log file in the repo folder `/nifi/user_scripts/` - - `log_invalid_records_to_file` = `True`, enable/disable logging errors to logfile with the above mentioned file name - - - IMPORTANT: all the original fields aside from doc id and text field that were returned by the ES search query will be placed in the "footer" dict key of each generated json record. - -3. `QueryNlpService-MedCAT-Bulk` - sends the bulk of documents in JSON format to MedCAT Service and receives back the annotations (bulk operation) - -document_annotation_id -4. `ParseJSON-ResponseContent-Bulk` - parses the received annotations, custom Jython script, with a few important properties: - - - `annotation_id_field` = `id` , this should not be changed, it is the internal annotation ID generated by MedCAT Service, check a sample MedCAT service response before changing! - - `document_id_field` = `docid`, the original unique document Id field, it is taken from the "footer" section of the response generated eariler by `ExecuteScript-ConvertRecordToMedCATinput` - - - `ignore_annotation_types` = `none` , comma separated values of annotation types to ignore - - `original_record_fields_to_include` = `none` , if you want more fields to be included you can add them here, separated by commas, these values are taken from the "footer" section of the Json previously generated before bulk processing - - -The script creates an unique annotation ID field by concatenating the document_id_field and annotation_id_field, resulting in something like : \${document_id_field}_${annotation_id_field}, this is stored as a flowfile attribute, for each record (which represents one annotation record!) under the name name 'document_annotation_id', it is later used in the 'PutElasticSearchJson' processor as the unique '_id' for ES, specified in the "Identifier Attribute" property of the processor. - - -Components (2) and (4) execute custom Jython scripts to parse the records (Flow Files). -These scripts are bundled with our custom Apache NiFi image and are available in [`../nifi/user_scripts`](https://github.com/CogStack/CogStack-NiFi/tree/main/nifi/user_scripts). - -The key component (3) is a generic HTTP client for communication with RESTful services. -It sends the JSON payload using `POST` method to `http://nlp-medcat-service-production:5000/api/process_bulk` endpoint. -MedCAT Service will process the multiple documents simultaneously, i.e. in bulk mode. -Please note that same instance of MedCAT Service can be used by different HTTP clients in multiple data pipelines. -See [the official MedCAT Service documentation](https://github.com/CogStack/MedCATservice/) for more information about the service and API use. -Figure below shows the configuration of the HTTP client. -![configure-medcart-w3](../_static/img/legacy_images/configure-medcat-w3.png) - -
- -## Indexing annotations by Elasticsearch -This example uses similar configuration for `PutElasticsearchRecord` component as before. -The annotations are now stored under `medical_reports_anns_medcat_medmen` index where the document identifier is specified by `document_annotation_id` field of the Flow File (it is being generated by the payload parsing script before). - -
- -## Annotating free-text documents (ES β†’ ES) -This workflow implements a modified NLP annotations ingestion pipeline based on the previous example. -It is assumed that now free-text documents were already ingested into Elasticsearch. -Moreover, here we are interested in extracting the NLP annotations only from documents matching a specific query for Elasticsearch. -As before, the annotations will be stored in Elasticsearch. -Figure below presents the full workflow. -![nifi-w4](../_static/img/legacy_images/nifi-workflow-w4.png) - -
- -## Reading documents from Elasticsearch -In this example, documents are read from the same Elasticsearch data store. -This is, the same one was used previously to store the documents, which were indexed under `medical_reports_text` index. -The documents are fetched matching an example query `{"query":{"match_all":{"document":"cancer"}}}`, i.e. the `document` field that will contain a `cancer` keyword. -Figure below presents the configuration of the Elasticsearch reader component `SearchElasticsearch`. -![nifi-w4](../_static/img/legacy_images/configure-es-reader-w4.png) - -
- -## Extracting NLP annotations from documents -Similarly, the annotations extraction is implemented by MedCAT Service that exposes RESful API for processing the documents. - -Since from Elasticsearch the records are received in JSON format, there is no need to perform parsing from AVRO format. -Hence, there are 3 NiFi components involved in this process: -1. `ExecuteGroovyScript-ConvertToNLP` - prepares the JSON payload for MedCAT Service, -2. `QueryNlpService-MedCAT-Bulk` - sends the bulk of documents in JSON format to MedCAT Service and receives back the annotations (bulk operation), -3. `ParseJSON-ResponseContent-Bulk` - parses the received annotations. - -Components (1) and (3) execute custom Groovy scripts to parse the records (Flow Files). -The configuration of HTTP client (2) remains the same as before. - -
- -## Indexing annotations by Elasticsearch -This example uses similar configuration for `PutElasticsearchRecord` component as before. -The annotations are now stored under `medical_reports_anns_medcat_medmen_cancer` index. - -
- -## Sending automated emails -You can automate email sending by using the PutEmail processor. You will need an Outlook or Gmail account to do this, of course, you can also use your own SMTP server if one is present. -For Gmail you would need to generate an app password : for a [personal accounts](https://myaccount.google.com/apppasswords), for [workspace accounts](https://support.google.com/a/answer/2956491?hl=en). -For Outlook, please see : for [organisations](https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/authenticated-client-smtp-submission) , for [personal accounts](https://support.microsoft.com/en-us/office/pop-imap-and-smtp-settings-for-outlook-com-d088b986-291d-42b8-9564-9c414e2aa040) . - -
- -## CogStack Common Schema conversion and ingestion - -As CogStack expands across institutions, there was a need to standardize the naming convention of the fields of most data structures we encountered. As such, we came up with our own strucutre based on the core fields found across organisations -, whilst respecting the OMOP data model. - -The main files responsible for the schema mapping are the following: - -`cogstack_common_schema_elasticsearch_index_mapping_template.json` - this is used by Elasticsearch, creates a mapping for a specified index. In general this does not need to be changed - -`cogstack_common_schema_mapping.json` - IMPORTANT: contains a JSON dict, here is where you can map your fields to the already predifined ones, as a sample we used the fields in the `medical_reports_text` DB and mapped them accordingly. - -`cogstack_common_schema_full.avsc` - this contains the Avro field specifications, including data types and so on. Usually this will not need changing. - -The flow is displayed in the image below. Split into two sections, it does the following: - - - create index mapping, from the mapping declared in `cogstack_common_schema_elasticsearch_index_mapping_template.json`, you can change the index name in the `PutElasticsearchJson` processor by changing the `Index` property. IMPORTANT: this only needs to be run once. - - - in the bottom second half of the image we convert the current fields gotten from the DB to our own schema mapping, the resulting record being in Avro format, we then ingest the record into ES. The only two processors that need changing here are `GenerateTableFetch` and `PutElasticsearchRecord`. - -![common-schema-workflow](../_static/img/legacy_images/cogstack_common_schema_full_workflow.png) - -## Ingesting raw files from disk with extra optional data - -NiFi template name: `Raw_file_read_from_disk_ocr_custom` - -Journey of this workflow: -1. execute python file that gets the record files from a folder which respects the yyyy/mm/dd pattern, each folder has a meta.csv file containing record data columns, - the free text is however stored in raw format however and needs to be OCR-ed, with the file name being the record ID from the meta.csv file. -We perform a lookup up and add the raw file content as a column to its corresponding record. -The result of the script is a JSON with all the records within a folder passted to STDOUT (only one flowfile can be generated per processor call since it is stdout). -2. we add the content-type as "application/json" to each flowfile, for completness. -3. we split the list of dictionaries so that we from one flowfile containing X records we will have a flowfile per record (required as for OCR-ing, we can only do one file per request). This step may require additional adjustment, the SplitJson processor has a text char limit (2mil) that it can split. If we go above that we can't split, so adjust the `output_batch_size` parameter of the `ExecuteProcess-getFilesFromDisk` processor, contained within the `Command Arguments` property, at the very end. -4. we convert the JSON output to AVRO for easier manipulation within the NiFi Jython script, which transfers the record's fields as attributes of a flowfile for future use -, and the designated binary data column is set as the content of the new flowfile, so that we send binary data to the ocr service. -5. perform the OCR -6. execute another custom python script to format the JSON response to contain the original record data fields too, at this stage the record is ready for ingestion into ES! - -Prerequisite if you want to test this template for testing, please run the following commands: -- `cd nifi/user_scripts/tests` -- `python3 generate_files.py` - -The above assumes that you already have the NiFi container running, the script just generates some sample files. - -## CogStack Cohort source file creation. - -Check the "CogStack_Cohort_create_source_docs" template, you will have to manually upload the xml if it is not already there (presuming you already have a working installation). - -This workflow will not work with sample data and annotations because there are not enough patients in the provided dataset. - -Prerequisites for this workflow: -1. make sure your concepts have been generated using a SNOMED model. -2. make sure you have enough patients (>1k) -3. you have the required fields in your patient records: age, ethnicity, date of death, date of birth, patient_id, doc_id, gender. -4. datetime fields must have the same format. - -The script used for this process is located here: `nifi/user_scripts/processors/cogstack_cohort_generate_data.py`. Please read all the info provided in the NiFi template. diff --git a/docs/nifi/development_guide.md b/docs/nifi/development_guide.md index 8d6af43a2..b63d73c0b 100644 --- a/docs/nifi/development_guide.md +++ b/docs/nifi/development_guide.md @@ -22,7 +22,7 @@ make init-security make start-nifi ``` -Then open NiFi at `https://localhost:8082`. +Then open NiFi through nginx at `https://localhost:8443`. ## Repository map diff --git a/docs/nifi/main.md b/docs/nifi/main.md index 9d8d98230..092a034c5 100644 --- a/docs/nifi/main.md +++ b/docs/nifi/main.md @@ -233,8 +233,7 @@ The key used ones are: ## User resources With our custom image there are bundled resources to get up and running example workflows. -Please see [current workflows](../deploy/workflows.md) and -[legacy workflows](../deploy/workflows_legacy.md) in the `deploy` docs for more details. +Please see [current workflows](../deploy/workflows.md) in the `deploy` docs for more details. ## Workflow templates @@ -281,7 +280,9 @@ To access the scheduling menu, right click on any processor > click configure > IMPORTANT: In general, if a process is left on the default `0 sec` run schedule, it will attempt to start to work as soon as flowfiles are in the queue, this can be a problem as, some workflows contain large amounts of flowfiles that may also contain a huge number of records, it can strain the system at times by constantly performing I/O operations (remember that flowfiles are stored both in memory and on disk), it is best to delay the running of tasks by setting the `Run Schedule` to anything but the default, this is especially useful for DB query processors (ExecuteSQLRecord or QueryDatabaseTable), in which the schedule should be set to an interval of minutes as queries may be very slow in executing. This setting will have a direct impact on CPU and memory usage. `Concurrent tasks` are a good way to keep things running in parallel if you feel that data is not being processed fast enough, just be aware that as stated above, one task represents one thread, and as mentioned in the `maximum thread count` section, it should not exceed the max number. -It should also be noted that a `processor` cannot have more `tasks` in current execution than the declared number `concurrent tasks`, if a task finishes its work you will notice that another will immidiately take it's place, of course, a new task may not be started if the current task has finished work and the timer is set on `run schedule` to be different than 0. +A processor cannot run more tasks concurrently than its configured +`Concurrent Tasks` value. When one task finishes, another can start immediately +if work is available and the run schedule permits it. ### Benchmarking diff --git a/mkdocs.yml b/mkdocs.yml index 83f798e67..3e62745c4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -42,10 +42,16 @@ nav: - πŸ“° News: news.md - πŸ“‹ Prerequisites: deploy/main.md - πŸ“¦ Deployment: deploy/deployment.md + - βš™οΈ Configuration: deploy/configuration.md - 🧩 Services: deploy/services.md - πŸ“› Troubleshooting: deploy/troubleshooting.md - - Workflows: deploy/workflows.md - - πŸ”€ Workflows: deploy/workflows_legacy.md + - πŸ”€ Workflows: + - Overview: deploy/workflows.md + - πŸ—„οΈ Database documents to OpenSearch: deploy/workflows/database_to_opensearch.md + - πŸ“„ Database documents via OCR: deploy/workflows/database_ocr_to_opensearch.md + - 🧠 MedCAT annotations to OpenSearch: deploy/workflows/medcat_annotations_to_opensearch.md + - πŸ“ Filesystem Parquet to OpenSearch: deploy/workflows/filesystem_parquet_to_opensearch.md + - ☁️ Azure Blob Parquet to OpenSearch: deploy/workflows/azure_blob_parquet_to_opensearch.md - πŸ’§ NiFi: - nifi/main.md - Development Guide: nifi/development_guide.md @@ -53,7 +59,6 @@ nav: - Processor Scripting: nifi/processor_scripting.md - User Scripts: nifi/user_scripts.md - Python Extensions: nifi/user_python_extensions.md - - Configuration: deploy/configuration.md - πŸ›‘οΈ Security: - security/main.md - πŸ›οΈ Certificates and Root CA: security/certificates.md