Skip to content

developmentSetup.md

Latest commit

 

History

History
200 lines (132 loc) · 8.36 KB

developmentSetup.md

File metadata and controls

200 lines (132 loc) · 8.36 KB
Raw

Setup for development

This guide aims to help first time contributors to setup and contribute to opencre.org. It assumes little/no prior development knowledge and an environment without any tools.

Table Of Contents

Prerequisites

opencre.org is a Python backend supported by a React frontend written using Typescript (or typed Javascript). In order to contribute code to opencre you need a system that is setup for python and Javascript development.

Linux

Assuming a Debian based system (e.g. ubuntu) you can install supporting tools with the following:

sudo apt update && sudo apt install -y curl git python3  build-essential curl libpq-dev python3-dev sqlite3

MacOS

Assuming the Homebrew package manager is installed, you can run the following.

brew install git python curl sqlite3 libpq make

The above will install

  • git , used for source control
  • python3 , used for backend development
  • build-essential , a package of useful development tools
  • curl , a utility that allows users to make network connections to remote servers, used for testing
  • sqlite3, a file based database used for development
  • make a build and execution automation tool

Language setup

Linux

We install python's utilities with

  • python3 -m pip install --upgrade pip pip is python's package manager
  • python3 -m pip install --upgrade virtualenv python virtual environment, used for python development
  • python3 -m pip install --upgrade setuptools python's setuptools is the package containing tooling for installing further python tools

MacOS

We install python's utilities with

  • brew install python3
  • brew install virtualenv

Then nodejs, used for Javascript development can be installed using the Node Version Manager as such:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash && nvm install --lts

Then we use npm nodejs, package manager to install yarn , a lightweight Javascript build tool.

npm install --global yarn

Database installation

OpenCRE uses SQLite for the default local development database and PostgreSQL for production. You do not need to install both databases for the standard contributor path; install PostgreSQL locally only when you are explicitly testing the PostgreSQL-backed workflow.

Linux

sudo apt install sqlite3 postgresql-client

MacOS

brew install sqlite3 postgresql

Development environment setup

OpenCRE uses git and follows the GitHub flow. To edit files it is suggested that a code editor such as Visual Studio Code is used. Visual Studio Code can be installed on Linux and MacOS.

Project Setup

  • Generate an SSH key pair and add it's public component to your Github account as described here.
  • Create a fork of the project following Github's instructions above.
  • Open a terminal
  • Clone your fork locally with git clone <the url of your fork starting with git@github.com>

OpenCRE depends on Makefiles to automate the setup and execution of several aspects. You can install the project by running make install.

Testing that everything works

Before running the end-to-end checks, start the local services and seed the fixture database. Then run both unit tests and end to end tests:

make start-containers
make e2e-db
make test
make e2e

A working setup should finish without a test failure and should make the local application available at http://localhost:5000.

Import the database

For local development, you do not need access to the OpenCRE Google Sheet. That spreadsheet is part of the maintainer workflow for bulk standards imports.

The supported contributor workflow is:

  1. create the local database schema
  2. sync the public OpenCRE graph from upstream
  3. import your own mappings locally through MyOpenCRE if needed

Run:

make migrate-upgrade
make upstream-sync

make upstream-sync downloads the OpenCRE graph from the public API into your local standards_cache.sqlite.

If you want to map your own standard locally, enable imports and use MyOpenCRE or the CSV import endpoint:

export CRE_ALLOW_IMPORT=true
make dev-flask

Then you can:

  • open the local MyOpenCRE UI
  • download a CSV template from GET /rest/v1/cre_csv
  • upload your CSV to POST /rest/v1/cre_csv_import after starting the local API
  • send the multipart file field as cre_csv

Example, once make dev-flask is running:

curl -X POST http://localhost:5000/rest/v1/cre_csv_import \
  -F cre_csv=@my_mappings.csv

Maintainer-only spreadsheet import example:

python cre.py --add --from_spreadsheet <google sheets url>

Running locally

First, start the required services (Redis and Neo4j) with:

make start-containers

You can run the backend with make dev-flask. At the time of writing the backend URL is http://localhost:5000 by default.

You can run the frontend with yarn start. This should open a browser tab at the application's front page and also automatically reload the page whenever changes are detected. At the time of writing the frontend URL is http://localhost:9001 by default.

Troubleshooting

  • virtualenv: command not found: install the virtualenv prerequisite, then rerun make install. Do not create a separate venv with a different tool.
  • make e2e fails before the tests start: make sure the local services are running and run make e2e-db first to seed the fixture database.
  • Docker command not found: use the Docker alternative from the main README, or install Docker before following that path.
  • Database confusion: the default local contributor path uses SQLite. Only add local PostgreSQL when you are testing the production-style database path.
  • Frontend dependency failures: the repository uses a Yarn v1 lockfile. Use Yarn Classic and keep the Node.js version consistent across contributors until the project adds an explicit Node.js engine constraint.

Regenerating all embeddings (smart extract / new model)

After changing embedding logic or models, wipe and rebuild stored vectors so the chatbot similarity search uses the new text:

# From repo root; uses standards_cache.sqlite by default or pass --cache_file
export OPENAI_API_KEY=...   # or GEMINI_API_KEY for Vertex
python cre.py --regenerate_embeddings --cache_file ./standards_cache.sqlite

This deletes every row in the embeddings table, then runs the same full pass as a cold import (Playwright fetch + optional smart excerpt + provider embed). It can take a long time and consumes API quota.

Chatbot and deep links

The chatbot matches your question to a standard node embedding, then answers using that node’s embeddings_content. When embeddings_url is set (e.g. OWASP AI Exchange with a #fragment), the API adds it as embeddingsUrl on the reference row alongside the unchanged catalog hyperlink; the UI shows a separate “Scoped source (embedding URL)” link. The LLM context includes an Embeddings_URL line for citations.

This is it, please follow the CONTRIBUTING guidelines while contributing and thank you for your interest in OpenCRE.