diff --git a/docs/DeveloperManuals/DeveloperSetup.md b/docs/DeveloperManuals/DeveloperSetup.md index 6d0bbd4bac8..47a284b25e9 100644 --- a/docs/DeveloperManuals/DeveloperSetup.md +++ b/docs/DeveloperManuals/DeveloperSetup.md @@ -10,6 +10,7 @@ sidebar_position: 1 - Docker v19.03.10+ - Golang v1.19+ +- Git 2.31+ (used by the `gitextractor` plugin) - GNU Make - Mac (Preinstalled) - Windows: [Download](http://gnuwin32.sourceforge.net/packages/make.htm) @@ -27,9 +28,8 @@ The following guide will walk through how to run DevLake's frontend (`config-ui` cd devlake ``` -2. Install dependencies for plugins: - - - [RefDiff](../Plugins/refdiff.md#development) +2. Make sure `git` (2.31 or newer) is on your `PATH`. The `gitextractor` plugin + uses it to clone and read repositories; no other plugin dependencies are required. 3. Install Go packages @@ -108,15 +108,9 @@ For provisioning, customizing, and creating dashboards, please refer to our [Gra ## Troubleshooting - Q: Running `make dev` yields error: `libgit2.so.1.3: cannot open share object file: No such file or directory` - - A: `libgit2.so.1.3` is required by the gitextractor plugin and should be . Make sure your program can find `libgit2.so.1.3`. `LD_LIBRARY_PATH` can be assigned like this if your `libgit2.so.1.3` is located at `/usr/local/lib`: + Q: Running `make dev` yields `git: executable file not found in $PATH` - ```sh - export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib - ``` - - Note that the version has to be pinned to 1.3.0. If you don't have it, you may need to build it manually with CMake from [source](https://github.com/libgit2/libgit2/releases/tag/v1.3.0). + A: The `gitextractor` plugin clones and reads repositories with the `git` command line. Install git 2.31 or newer and make sure it is on your `PATH`. ## Compiling diff --git a/docs/Plugins/gitextractor.md b/docs/Plugins/gitextractor.md index b4e4a206fb2..0b63249a410 100644 --- a/docs/Plugins/gitextractor.md +++ b/docs/Plugins/gitextractor.md @@ -54,84 +54,33 @@ curl --location --request POST 'localhost:8080/pipelines' \ You call also run this plugin in a standalone mode without any DevLake service running using the following command: ``` -go run plugins/gitextractor/main.go -url https://github.com/merico-dev/lake.git -id github:GithubRepo:384111310 -db "merico:merico@tcp(127.0.0.1:3306)/lake?charset=utf8mb4&parseTime=True" +cd backend +go run ./plugins/gitextractor -url https://github.com/apache/devlake.git -repoId github:GithubRepo:384111310 ``` -For more options (e.g., saving to a csv file instead of a db), please read `plugins/gitextractor/main.go`. +It uses the database configured in `.env` (`DB_URL`). + +For more options (e.g., `skipCommitStat`, `skipCommitFiles`), please read `plugins/gitextractor/gitextractor.go`. ## Development -This plugin depends on `libgit2`, you need to install version 1.3.0 to run and debug this plugin on your local -machine. +This plugin reads repositories by running `git` commands against a local clone, +so it requires **git 2.31 or newer** where DevLake runs. libgit2, cgo and a C +toolchain are no longer needed. ### Linux ``` -1. require cmake [ubuntu] -apt install cmake -y -[centos] -yum install cmake -y - -2. compiling -git clone -b v1.3.0 https://github.com/libgit2/libgit2.git && cd libgit2 -mkdir build && cd build && cmake .. -make && make install - -3.PKG_CONFIG and LD_LIBRARY_PATH +apt install git -y [centos] -export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib64:/usr/local/lib64/pkgconfig -export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib64 -[ubuntu] -export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig -export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib +yum install git -y ``` -#### Troubleshooting (Linux) - -> Q: # pkg-config --cflags -- libgit2 Package libgit2 was not found in the pkg-config search path. -> Perhaps you should add the directory containing `libgit2.pc` to the PKG_CONFIG_PATH environment variable -> No package 'libgit2' found pkg-config: exit status 1 - -> A: -> Make sure your pkg config path covers the installation: -> if your libgit2.pc in `/usr/local/lib64/pkgconfig`(like centos) -> -> `export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib64:/usr/local/lib64/pkgconfig` -> -> else if your libgit2.pc in `/usr/local/lib/pkgconfig`(like ubuntu) -> -> `export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig` -> -> else consider install pkgconfig or rebuild the libgit2 - ### MacOS -NOTE: **Do NOT** install libgit2 via `MacPorts` or `homebrew`, install from source instead. - ``` -brew install cmake -git clone https://github.com/libgit2/libgit2.git -cd libgit2 -git checkout v1.3.0 -mkdir build -cd build -cmake .. -make -make install +brew install git ``` -#### Troubleshooting (MacOS) - -> Q: I got an error saying: `pkg-config: exec: "pkg-config": executable file not found in $PATH` - -> A: -> -> 1. Make sure you have pkg-config installed: -> -> `brew install pkg-config` -> -> 2. Make sure your pkg config path covers the installation: -> `export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig` -


diff --git a/docs/Plugins/refdiff.md b/docs/Plugins/refdiff.md index 2394c73c651..101031ef2a2 100644 --- a/docs/Plugins/refdiff.md +++ b/docs/Plugins/refdiff.md @@ -124,9 +124,4 @@ curl 'http://localhost:8080/pipelines' \ }' ``` -## Development - -This plugin depends on `libgit2`, you need to install version 1.3.0 in order to run and debug this plugin on your local -machine. [Click here](./gitextractor.md#Development) for a brief guide. -


diff --git a/docs/Troubleshooting/Configuration.md b/docs/Troubleshooting/Configuration.md index 4f3e9b846fc..fe31a714375 100644 --- a/docs/Troubleshooting/Configuration.md +++ b/docs/Troubleshooting/Configuration.md @@ -19,7 +19,7 @@ description: > There might be two problems when trying to collect data from a private GitLab server with a self-signed certificate: 1. "Test Connection" error. This can be solved by setting the environment variable `IN_SECURE_SKIP_VERIFY=true` for the `devlake` container -2. "GitExtractor" fails to clone the repository due to certificate verification, sadly, neither gogit nor git2go we are using supports insecure HTTPS. +2. "GitExtractor" fails to clone the repository due to certificate verification. Setting `IN_SECURE_SKIP_VERIFY=true` also disables TLS verification for the git clone (`GIT_SSL_NO_VERIFY=true`). A better approach would be adding your root CA to the `devlake` container: