Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 5 additions & 11 deletions docs/DeveloperManuals/DeveloperSetup.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ sidebar_position: 1

- <a href="https://docs.docker.com/get-docker" target="_blank">Docker v19.03.10+</a>
- <a href="https://golang.org/doc/install" target="_blank">Golang v1.19+</a>
- <a href="https://git-scm.com/downloads" target="_blank">Git 2.31+</a> (used by the `gitextractor` plugin)
- <a href="https://www.gnu.org/software/make/" target="_blank">GNU Make</a>
- Mac (Preinstalled)
- Windows: [Download](http://gnuwin32.sourceforge.net/packages/make.htm)
Expand All @@ -27,9 +28,8 @@ The following guide will walk through how to run DevLake's frontend (`config-ui`
cd devlake
```

2. Install dependencies for plugins:

- [RefDiff](../Plugins/refdiff.md#development)
2. Make sure `git` (2.31 or newer) is on your `PATH`. The `gitextractor` plugin
uses it to clone and read repositories; no other plugin dependencies are required.

3. Install Go packages

Expand Down Expand Up @@ -108,15 +108,9 @@ For provisioning, customizing, and creating dashboards, please refer to our [Gra
## Troubleshooting


Q: Running `make dev` yields error: `libgit2.so.1.3: cannot open share object file: No such file or directory`

A: `libgit2.so.1.3` is required by the gitextractor plugin and should be . Make sure your program can find `libgit2.so.1.3`. `LD_LIBRARY_PATH` can be assigned like this if your `libgit2.so.1.3` is located at `/usr/local/lib`:
Q: Running `make dev` yields `git: executable file not found in $PATH`

```sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
```

Note that the version has to be pinned to 1.3.0. If you don't have it, you may need to build it manually with CMake from [source](https://github.com/libgit2/libgit2/releases/tag/v1.3.0).
A: The `gitextractor` plugin clones and reads repositories with the `git` command line. Install git 2.31 or newer and make sure it is on your `PATH`.


## Compiling
Expand Down
73 changes: 11 additions & 62 deletions docs/Plugins/gitextractor.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,84 +54,33 @@ curl --location --request POST 'localhost:8080/pipelines' \
You call also run this plugin in a standalone mode without any DevLake service running using the following command:

```
go run plugins/gitextractor/main.go -url https://github.com/merico-dev/lake.git -id github:GithubRepo:384111310 -db "merico:merico@tcp(127.0.0.1:3306)/lake?charset=utf8mb4&parseTime=True"
cd backend
go run ./plugins/gitextractor -url https://github.com/apache/devlake.git -repoId github:GithubRepo:384111310
```

For more options (e.g., saving to a csv file instead of a db), please read `plugins/gitextractor/main.go`.
It uses the database configured in `.env` (`DB_URL`).

For more options (e.g., `skipCommitStat`, `skipCommitFiles`), please read `plugins/gitextractor/gitextractor.go`.

## Development

This plugin depends on `libgit2`, you need to install version 1.3.0 to run and debug this plugin on your local
machine.
This plugin reads repositories by running `git` commands against a local clone,
so it requires **git 2.31 or newer** where DevLake runs. libgit2, cgo and a C
toolchain are no longer needed.

### Linux

```
1. require cmake
[ubuntu]
apt install cmake -y
[centos]
yum install cmake -y

2. compiling
git clone -b v1.3.0 https://github.com/libgit2/libgit2.git && cd libgit2
mkdir build && cd build && cmake ..
make && make install

3.PKG_CONFIG and LD_LIBRARY_PATH
apt install git -y
[centos]
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib64:/usr/local/lib64/pkgconfig
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib64
[ubuntu]
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
yum install git -y
```

#### Troubleshooting (Linux)

> Q: # pkg-config --cflags -- libgit2 Package libgit2 was not found in the pkg-config search path.
> Perhaps you should add the directory containing `libgit2.pc` to the PKG_CONFIG_PATH environment variable
> No package 'libgit2' found pkg-config: exit status 1

> A:
> Make sure your pkg config path covers the installation:
> if your libgit2.pc in `/usr/local/lib64/pkgconfig`(like centos)
>
> `export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib64:/usr/local/lib64/pkgconfig`
>
> else if your libgit2.pc in `/usr/local/lib/pkgconfig`(like ubuntu)
>
> `export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig`
>
> else consider install pkgconfig or rebuild the libgit2

### MacOS

NOTE: **Do NOT** install libgit2 via `MacPorts` or `homebrew`, install from source instead.

```
brew install cmake
git clone https://github.com/libgit2/libgit2.git
cd libgit2
git checkout v1.3.0
mkdir build
cd build
cmake ..
make
make install
brew install git
```

#### Troubleshooting (MacOS)

> Q: I got an error saying: `pkg-config: exec: "pkg-config": executable file not found in $PATH`

> A:
>
> 1. Make sure you have pkg-config installed:
>
> `brew install pkg-config`
>
> 2. Make sure your pkg config path covers the installation:
> `export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig`

<br/><br/><br/>
5 changes: 0 additions & 5 deletions docs/Plugins/refdiff.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,4 @@ curl 'http://localhost:8080/pipelines' \
}'
```

## Development

This plugin depends on `libgit2`, you need to install version 1.3.0 in order to run and debug this plugin on your local
machine. [Click here](./gitextractor.md#Development) for a brief guide.

<br/><br/><br/>
2 changes: 1 addition & 1 deletion docs/Troubleshooting/Configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ description: >
There might be two problems when trying to collect data from a private GitLab server with a self-signed certificate:

1. "Test Connection" error. This can be solved by setting the environment variable `IN_SECURE_SKIP_VERIFY=true` for the `devlake` container
2. "GitExtractor" fails to clone the repository due to certificate verification, sadly, neither gogit nor git2go we are using supports insecure HTTPS.
2. "GitExtractor" fails to clone the repository due to certificate verification. Setting `IN_SECURE_SKIP_VERIFY=true` also disables TLS verification for the git clone (`GIT_SSL_NO_VERIFY=true`).

A better approach would be adding your root CA to the `devlake` container:

Expand Down
Loading