Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .github/workflows/build-and-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ on:
branch:
type: string
required: false
go-modules:
type: string
required: true

jobs:
build-and-test:
Expand All @@ -22,7 +25,7 @@ jobs:
matrix:
#FIXME os: [ubuntu-latest, macOS-latest, windows-latest]
os: [ubuntu-latest]
go-module: [utils]
go-module: ${{ fromJson(inputs.go-modules) }}
multi-platform:
- ${{ github.ref == 'refs/heads/main' || github.ref == 'refs/heads/master' || github.event_name == 'workflow_dispatch' }}
exclude:
Expand Down Expand Up @@ -83,12 +86,15 @@ jobs:
version: latest
working-directory: ${{ matrix.go-module }}
args: "--verbose --output.text.print-issued-lines --output.text.print-linter-name --output.tab.colors --max-issues-per-linter 0 --max-same-issues 0"
- name: Derive artifact-safe module label
shell: bash
run: echo "MODULE_ARTIFACT_LABEL=${{ matrix.go-module }}" | tr '/' '-' >> "$GITHUB_ENV"
- name: Set up gotestsum
uses: ARM-software/golang-utils/actions/setup-gotestsum@v1.172.0
- name: Test ${{ matrix.go-module }} on ${{ matrix.os }}
uses: ARM-software/golang-utils/actions/test-go-modules@v1.172.0
with:
modules: ./${{ matrix.go-module }}
build-tags: ""
artifact-prefix: "${{ matrix.go-module }}-${{ matrix.os }}-"
artifact-prefix: "${{ env.MODULE_ARTIFACT_LABEL }}-${{ matrix.os }}-"
report-name: "${{ matrix.go-module }} tests on ${{ matrix.os }}"
23 changes: 22 additions & 1 deletion .github/workflows/update-utils.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: Update Library
# Set environment variables available in all jobs and steps
env:
python_version: "3.11"
go_module: "utils"
go_version: "1.27.3"

on:
workflow_call:
Expand Down Expand Up @@ -52,11 +52,32 @@ jobs:
dockerfile: Dockerfile.utils
branch: ${{ inputs.branch || github.head_ref || github.ref }}

discover-go-modules:
name: Discover Go modules
needs:
- update-go
runs-on: ubuntu-latest
outputs:
modules: ${{ steps.detect.outputs.modules-json }}
steps:
- uses: actions/checkout@v7.0.1
with:
ref: ${{ inputs.branch || github.head_ref || github.ref }}
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.go_version }}
- name: Detect Go modules
id: detect
uses: ARM-software/golang-utils/actions/detect-go-modules@v1.172.0

build-and-test:
name: Build and Test
needs:
- update-go
- discover-go-modules
uses: ./.github/workflows/build-and-test.yml
secrets: inherit
with:
branch: ${{ inputs.branch || github.head_ref || github.ref }}
go-modules: ${{ needs.discover-go-modules.outputs.modules }}
38 changes: 27 additions & 11 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,17 @@ Since this is a Go library, there is no application to run.

### Running tests

To run all tests:
To run all tests, execute `go test ./...` from each module directory.

To discover the workspace modules from the repository root:
```bash
go test ./...
go list -m -json
```

To refresh workspace and module discovery metadata after layout changes:
```bash
go list -m -json
go work sync
```

### Mocks
Expand All @@ -35,8 +43,7 @@ It is up to the individual developer to ensure their change complies with this.
Static analysis tools and linters are run as part of CI.
They come from [golangci-lint](https://golangci-lint.run/). To run this locally:
```bash
# Must be in a directory with a go.mod file
cd <directory_with_go_module>
# Run golangci-lint from each Go module directory.
golangci-lint run ./...
```

Expand Down Expand Up @@ -84,16 +91,25 @@ The version number will be automatically calculated, based on the news files.

# Detecting secrets

So that no secrets are committed back to the repository, a combination of two tools are run in CI:
- [GitLeaks]() : Scans the git history for usual secrets (e.g. AWS keys, etc.)
- [detect-secrets](https://github.com/Yelp/detect-secrets): Scans only the current state of the repository for anything which can look like secrets (strings with high entropy)
So that no secrets are committed back to the repository, CI runs a combination of two tools:
- `cd-detect-secrets` from [continuous-delivery-scripts](https://github.com/ARMmbed/continuous-delivery-scripts): checks the current repository content for new secret findings against the accepted registry
- [GitLeaks](): scans the git history for usual secrets (for example AWS keys)

For the latter, False positive keys are stored in the [baseline](./.secrets.baseline) which `detect-secrets` checks against when it runs
To run the repository secret check locally:
```bash
python -m pip install --upgrade continuous-delivery-scripts
cd-detect-secrets
```

Accepted findings are stored in the [baseline](./.secrets.baseline), which `cd-detect-secrets` uses when it runs.

## Baseline & False positives

To flag individual false positives add comment `# pragma: allowlist secret` to line with secret
To flag individual false positives add comment `# pragma: allowlist secret` to the line with the finding.

To add all suspected secrets in the repository (excluding ones with an allow secret comment), run `detect-secrets scan --all-files --exclude-files '.*go\.sum$' --exclude-files '.*\.html$' --exclude-files '.*\.properties$' --exclude-files 'ci.yml' --exclude-files '\.git' > .secrets.baseline`
To update the accepted registry after review, use:
```bash
cd-record-secrets --registry-file .secrets.baseline
```

If on Windows: then change the encoding of the .secrets.baseline file to UTF-8 then convert all `\` to `/` in the .secrets.baseline file
If you edit `.secrets.baseline` manually on Windows, keep it UTF-8 encoded and convert `\` path separators to `/`.
17 changes: 12 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,13 @@ Few helpers may be themselves leveraging 3rd party libraries.

## Using this library

To use this library, add the following line to your `go.mod`:
Each importable package under `utils` is now its own Go module. Add the module that matches the package you import, for example:
```
require (
github.com/ARM-software/embedded-development-services-client-utils/utils latest
...
)
go get github.com/ARM-software/embedded-development-services-client-utils/utils/messages@latest
```

The top-level `github.com/ARM-software/embedded-development-services-client-utils/utils` module remains as a compatibility umbrella for workspace and tooling use.


## Releases

Expand All @@ -52,6 +51,14 @@ The follow described the major aspects of the project structure:
- For a list of known issues and possible workarounds, please see [Known Issues](KNOWN_ISSUES.md).
- To raise a defect or enhancement please use [GitHub Issues](https://github.com/ARM-software/embedded-development-services-client-utils/issues) or [GitHub discussions](https://github.com/ARM-software/embedded-development-services-client-utils/discussions).

## Secrets Checks

Use [continuous-delivery-scripts](https://github.com/ARMmbed/continuous-delivery-scripts) to run the repository secret checks locally:
```bash
python -m pip install --upgrade continuous-delivery-scripts
cd-detect-secrets
```

## Contributing

- We are committed to fostering a welcoming community, please see our
Expand Down
1 change: 1 addition & 0 deletions changes/202611181600.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added workspace-aware multi-module support so each importable package under `utils` can be consumed and tested as its own Go module.
1 change: 1 addition & 0 deletions changes/202611181601.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Refactored CI module discovery to detect workspace modules once and pass that shared list through the reusable build-and-test workflow.
20 changes: 19 additions & 1 deletion go.work
Original file line number Diff line number Diff line change
@@ -1,3 +1,21 @@
go 1.27

use ./utils
use (
./utils
./utils/api
./utils/artefacts
./utils/cache
./utils/client
./utils/errors
./utils/job
./utils/job/jobtest
./utils/links
./utils/links/linkstest
./utils/logging
./utils/messages
./utils/mocks
./utils/pagination
./utils/resource
./utils/resource/resourcetests
./utils/store
)
7 changes: 4 additions & 3 deletions go.work.sum
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@ github.com/go-git/go-billy/v5 v5.6.2 h1:6Q86EsPXMa7c3YZ3aLAQsMA0VlWmy43r6FHqa/UN
github.com/go-git/go-billy/v5 v5.6.2/go.mod h1:rcFC2rAsp/erv7CMz9GczHcuD0D32fWzH+MJAU+jaUU=
github.com/go-git/go-git/v5 v5.16.4 h1:7ajIEZHZJULcyJebDLo99bGgS0jRrOxzZG4uCk2Yb2Y=
github.com/go-git/go-git/v5 v5.16.4/go.mod h1:4Ge4alE/5gPs30F2H1esi2gPd69R0C39lolkucHBOp8=
github.com/godbus/dbus v4.1.0+incompatible h1:WqqLRTsQic3apZUK9qC5sGNfXthmPXzUZ7nQPrNITa4=
github.com/golang/glog v0.0.0-20210429001901-424d2337a529 h1:2voWjNECnrZRbfwXxHB1/j8wa6xdKn85B5NzgVL/pTU=
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 h1:f+oWsMOmNPc8JmEHVZIycC7hBoQxHH9pNKQORJNozsQ=
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8/go.mod h1:wcDNUvekVysuuOpQKo3191zZyTpiI6se1N1ULghS0sw=
Expand Down Expand Up @@ -62,7 +61,6 @@ github.com/pascaldekloe/name v1.0.0/go.mod h1:Z//MfYJnH4jVpQ9wkclwu2I2MkHmXTlT9w
github.com/pjbgf/sha1cd v0.3.2 h1:a9wb0bp1oC2TGwStyn0Umc/IGKQnEgF0vVaZ8QF8eo4=
github.com/pjbgf/sha1cd v0.3.2/go.mod h1:zQWigSxVmsHEZow5qaLtPYxpcKMMQpa09ixqBxuCS6A=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pquerna/ffjson v0.0.0-20190930134022-aa0246cd15f7 h1:xoIK0ctDddBMnc74udxJYBqlo9Ylnsp1waqjLsnef20=
Expand Down Expand Up @@ -92,8 +90,12 @@ github.com/yuin/goldmark v1.4.13 h1:fVcFKWvrslecOb/tg+Cc05dkeYx540o0FuFt3nUVDoE=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 h1:nwGZBCt+FnXUrGsj5vjzAsEmkcaFvd82BbOjECiFYZc=
golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57/go.mod h1:3AWMyWHS+caVoiEXpiq6+tzKA40J4vQT3MYr80ZtQpc=
golang.org/x/telemetry v0.0.0-20260924152758-ed294f943157 h1:DftxASHKl6JolFPqoN+2WrFO53JvkHwRXvFqf8gsivA=
Expand All @@ -119,6 +121,5 @@ gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkep
gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME=
gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI=
gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
sigs.k8s.io/yaml v1.6.0 h1:G8fkbMSAFqgEFgh4b1wmtzDnioxFCUgTZhlbj5P9QYs=
sigs.k8s.io/yaml v1.6.0/go.mod h1:796bPqUfzR/0jLAl6XjHl3Ck7MiyVv8dbTdyT3/pMf4=
68 changes: 68 additions & 0 deletions utils/api/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
module github.com/ARM-software/embedded-development-services-client-utils/utils/api

go 1.27

replace github.com/ARM-software/embedded-development-services-client-utils/utils/errors => ../errors

replace github.com/ARM-software/embedded-development-services-client-utils/utils/cache => ../cache

replace github.com/ARM-software/embedded-development-services-client-utils/utils/links => ../links

replace github.com/ARM-software/embedded-development-services-client-utils/utils/artefacts => ../artefacts

replace github.com/ARM-software/embedded-development-services-client-utils/utils/client => ../client

replace github.com/ARM-software/embedded-development-services-client-utils/utils/job => ../job

replace github.com/ARM-software/embedded-development-services-client-utils/utils/job/jobtest => ../job/jobtest

replace github.com/ARM-software/embedded-development-services-client-utils/utils/links/linkstest => ../links/linkstest

replace github.com/ARM-software/embedded-development-services-client-utils/utils/logging => ../logging

replace github.com/ARM-software/embedded-development-services-client-utils/utils/messages => ../messages

replace github.com/ARM-software/embedded-development-services-client-utils/utils/mocks => ../mocks

replace github.com/ARM-software/embedded-development-services-client-utils/utils/pagination => ../pagination

replace github.com/ARM-software/embedded-development-services-client-utils/utils/resource => ../resource

replace github.com/ARM-software/embedded-development-services-client-utils/utils/resource/resourcetests => ../resource/resourcetests

replace github.com/ARM-software/embedded-development-services-client-utils/utils/store => ../store

require (
github.com/ARM-software/embedded-development-services-client-utils/utils/errors v0.0.0-00010101000000-000000000000
github.com/ARM-software/embedded-development-services-client/client v1.109.1
github.com/ARM-software/golang-utils/utils/commonerrors v1.172.0
github.com/ARM-software/golang-utils/utils/commonerrors/errortest v1.172.0
github.com/ARM-software/golang-utils/utils/field v1.172.0
github.com/ARM-software/golang-utils/utils/http v1.172.0
github.com/go-faker/faker/v4 v4.12.0
github.com/stretchr/testify v1.12.1
)

require (
github.com/ARM-software/golang-utils/utils/collection v1.172.0 // indirect
github.com/ARM-software/golang-utils/utils/parallelisation v1.172.0 // indirect
github.com/ARM-software/golang-utils/utils/ptr v1.172.0 // indirect
github.com/ARM-software/golang-utils/utils/reflection v1.172.0 // indirect
github.com/ARM-software/golang-utils/utils/safecast v1.172.0 // indirect
github.com/ARM-software/golang-utils/utils/safeio v1.172.0 // indirect
github.com/ARM-software/golang-utils/utils/value v1.172.0 // indirect
github.com/deckarep/golang-set/v2 v2.9.0 // indirect
github.com/dolmen-go/contextio v1.0.0 // indirect
github.com/hashicorp/errwrap v1.0.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/mailru/easyjson v0.9.2 // indirect
github.com/perimeterx/marshmallow v1.1.5 // indirect
github.com/petermattis/goid v0.0.0-20250813065127-a731cc31b4fe // indirect
github.com/sasha-s/go-deadlock v0.3.9 // indirect
go.mongodb.org/mongo-driver v1.17.7 // indirect
go.uber.org/atomic v1.12.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/sync v0.24.0 // indirect
golang.org/x/text v0.43.0 // indirect
)
Loading
Loading