Improve data model and the loading service

@wurstbrot

Data model

The current data model and data flow, reads the generated.yaml from the DSOMM web server (either the public dsomm.owasp.org. or a private one).

The file contains a list of activity info (all dimensions, subdimensions, activity description, , as well as each team's progress of implementations and evidence.

The generated.yaml if either copied from DSOMM-data or generated from an internal process in the organisation.

A user can update the team's progress in the Circular Heatmap. Then (more or less) a copy of the generated.yaml is stored in the browser's localStorage.

Drawbacks

The current model is storing activity description data in localStorage. And mixes "slowly changing information" with "live data" (i.e. activity + team status).

The current load process is duplicated in the matrix, activity description, circular heatmap, dependency graph, and mapping component. This leads to unnecessary load time, and complex maintenance.

Suggestion

1) Create a central load service

I suggest we create a new service that loads the yaml files, and merging this with progress data stored in localStorage.

This service will handle the race conditions, and let the other components just consume the already loaded data.

Benefit: Cleaner architecture, easier debugging, speed improvements and increased robustness when navigating in DSOMM.

2) Split the Activity info and Team's progress

I suggest we split the current generated.yaml file into:

activity info
team progress

(I'm completely open to name suggestions.)

The Activity Info contains the (fairly) static information contained in today's activity object.
The Team Progress contains team name, teamsImplemented and teamsEvidence. The Team Progress have a yaml file on the server, where any centralized progress may be updated.
The Team Progress will be stored in localStorage if any progress is changed in the browser. It will be merged with any updates from the server file when loading.
The Team Progress may be exported, as today, to update the server file.

This approach will also allow a user to change teams' names even for a public site such as dsomm.owasp.org.

Mermaid diagram

Here is a draft of the suggestion.

graph LR


DsommAct(activities.yaml)

DataAct --> DsommAct
subgraph "GitHub DSOMM-data"
    DataMeta(meta.yaml)

    Build.yaml --> DataAct
    Deploy.yaml --> DataAct
    PathManagement.yaml --> DataAct
    Other("... .yaml") --> DataAct
end

DataMeta --> DsommMeta
DataAct(activities.yaml)
DsommCustAct(custom-activities.yml)

subgraph DSOMM Web server
    DsommMeta(meta.yaml)
    DsommAct
    DsommCustAct
end

DsommMeta --> BrowserTeams("Teams")
DsommAct --> BrowserAct("Activities")
DsommCustAct --> BrowserCustAct(custom-activities.yml)
subgraph Browser 
    BrowserTeams -.-> BrowserProgress
    BrowserAct -.-> BrowserProgress
    BrowserCustAct-.-> BrowserProgress
    BrowserProgress("TeamProgress")
end
subgraph Browser localStorage
    BrowserTeams <--> LocalTeams("Teams")
    BrowserProgress <--> LocalProgress("TeamProgress")
end

(Mermaid is great. But has some limits. The following draft sketch should have a TeamProgress in the DSOMM Webserver, but then the drawing went bonkers, so I left it out.)

Comments?

@wurstbrot and @0x41head: Any comments on this?

I see that @yashchauhan4579 created a PR #379 just now, to upgrade Angular, aiming for Angular 19, down the line. (👍 🙌 🏆 )

They published that while I was writing this. I think it would be unwise to run these two initiatives in parallel. Even if they have different focus, there will be clashes. Nevertheless, I think it is still worthwhile discussing this idea, and planning the future.

@vbakke

Thanks, @vbakke! Appreciate your support! 😊 Looking forward to the next steps.

@wurstbrot

@wurstbrot: In assets\YAML\ there is currently a teams.yaml as well as the teams being defined in meta.yaml?

What was the reason to move the team definition into the meta file? Is that reason still valid?

To me, it also makes sense to have the teams defined in a separate teams.yaml

@ptechofficial

In GSoC with @ptechofficial , we first had teams in teams.yaml and afterwards moved teams to meta.yaml.
I think it makes sense to go back to teams.yaml.

@wurstbrot

My first draft for structuring the yaml files as follows:

─ assets
  └─ YAML
     ├─ meta.yaml 
     ├─ teams.yaml
     ├─ teamprogress.yaml
     ├─ default
     |    ├─ activities.yaml
     └─ custom
           └─ custom-activities.yaml

The teamprogress.yaml will start empty, but can be replaced with an exported file from the UI. Will contain data for both teamsImplemeneted and teamsEvidence from today's setup.

The meta.yaml will contain "links" to the activity files. Files in the default folder comes from DSOMM-data, and can be updated from DSOMM-data, without causing any harm to local customizations, nor any team progress.

The customs folder if for local customization, that can override the default DSOMM files. (It should contain a "dummy file" with examples.) A custom activity file should allow an ignore attribute, to "remove" an unwanted default activity.

meta.yaml

as today, but adds:

    checkForDsommUpdates: true

    $teamsRef: teams.yaml
    $progressRef: teamprogress.yaml
    $activityRef:
        - default/activities.yaml 
        - custom/custom-activities.yaml  # For customizing your own activities

Not sure about the naming convention $teamRef. It sort of plays on the yaml $ref feature. But does not replace the content, as the yaml spec says.

The order of $activityRef dictates the priority. Last entry takes priority over earlier files.

Updates from DSOMM

I suggest that the DSOMM client checks github for any new releases of the activities file (currently named generated.yaml).
This must of course be possible to turn off. I propose checkForDsommUpdates in the meta.yaml for controlling the behaviour.

The https://api.github.com/repos/devsecopsmaturitymodel/DevSecOps-MaturityModel-data/releases/latest contains the latest release. The public version can be verified against local version.
The user gets choices in the browser, if a new version is released:

Remind me again next time
Don't remind again for this update
Don't remind me again for any update

Comments

@wurstbrot or @0x41head: I'd like to hear any comments, suggestions or resistance to this.

@wurstbrot

After some tinkering with the meta.yaml I have implmented the yaml $ref feature, and suggest:

checkForDsommUpdates: true
teams: 
  $ref: 'default/teams.yaml#/teams'
teamGroups:
  $ref: 'default/teams.yaml#/teamGroups'

teamProgressFile: 'teamprogress.yaml' 
activityFiles:
    - default/activities.yaml 
    - custom/custom-activities.yaml  # For customizing your own activities

Customized activity files may include an ignore: true attribute to "remove" an activity (or dimension) from the standard DSOMM list.

And also, I do not see the purpose of the strings property in the DSOM meta.yaml. Are these remnants from thr DSOMM-Data project, @wurstbrot ?

That is correct, they would be better placed in the data project.

@0x41head

@0x41head: Do you have any comments on: https://github.com/vbakke/DevSecOps-MaturityModel/tree/feat/central-loader

(Before I refactor how the matrix, circular heatmap, etc load and display its data : )

@wurstbrot

Terminology
My gut feeling tells me that DSOMM is focusing more on the 'Sub-Dimension' level, than the 'Dimension' level.

What do you feel about renaming the two concepts to 'Dimension' and 'Category'?

Thus:

'Dimension' -> 'Category'
'Sub-Dimension' -> 'Dimension'

(At least my brain always starts sidetracking, following associations as what a subdimension is it sub to? The term 'Category' is more clearly a group of other items. And I find it easier to talk about the 'Design dimension', rather than the 'Design subdimension'.)

Any thoughts from anyone?
@wurstbrot?

@wurstbrot

@wurstbrot, is renaming okay for you?

@wurstbrot

@wurstbrot or @0x41head: I'd love to hear comments about the refactoring.

I have published a fork of the the latest update in feat/central-loader-matrix on: https://dsomm.pages.dev/ as "experimental edition".

Current status:

new data-loader-service and yaml-loader
matrix page is using new format (and IMHO is simplified)
improved feedback if yaml files don't pass valiudation

Next up is the activity details page, and then circular heatmap.

Filters
My intention was not to modify the UI. But I ended up with a modification of how the filters work for the Matrix.

I prefer the standard BI way of selecting filters:

no filters => show everything
some filters selected: show these filters only

Please let me know if you object.

@vbakke

Hi @vbakke,
dimension and sub-dimension are scientific terms used in maturity models for categories and sub-categories.
I am not convinced that a renaming is needed. If a renaming would happen, there shouldn't be a shift from dimension to sub-dimensions being called "dimensions" (which has been used beforehand).

I didn't go through the hole code, but the new architecture looks good in general. We can discuss it in a PR.

Flipping filters is great.

Why did you change the background color from grey to yellowish?

@wurstbrot

Thank you @wurstbrot. The yellowish tint is just in the experimental edition, so I don't get confused as to which I am looking at.

Thank you for having a look at the overall architecture. PR will come. But all the rendering gets improved as a result unifying the data structure. So it becomes a quite large PR. And one cannot split it, due to the inherent dependencies.

I don't know any maturity models using the term 'sub-dimensions'. The ones I looked up, uses different terminology on the various levels of the hierarchical break-down. (Such as 'Business function', 'Security practices', and 'Streams' in SAMM.)

The term 'dimensions', is fine used in a DSOMM context. I just don't see the term "Build and Deployment" as one dimension, but rather a group of dimensions. (The 'and' sort of implies that : ) While 'Build' I find more natural being one dimension.

I can revert back to using 'dimensions' and 'sub-dimensions' if you'd like. But I would applaud having terms that indicated the level in the hierarchy, without having to use the word 'sub'. 'Sub' is understandable, but not optimal.

not stale, just tedious

Status update: Working on editing Teams' names in-browser

First draft for how to edit teams and groups in browser published on: https://dsomm.pages.dev/teams

Published basic KPIs for teams on experimental edition: https://dsomm.pages.dev/teams

Finally got the linting and unit tests done.

~~Issue pushed to branch v4~~

Created PR #417

The DSOMM v4 is getting closer to release. The activity desctiption has gotten a revamp:

And a Settings page is underway...

https://dsomm.pages.dev/

This issue has been automatically marked as stale because it has not had recent activity. 📆 It will be closed automatically in one week if no further activity occurs.

Brushing up on the Markdown files now. Getting closer to release...

Finally closed with the v3.31

Navigation Menu

Repository navigation

Improve data model and the loading service #380

Data model

Drawbacks

Suggestion

1) Create a central load service

2) Split the Activity info and Team's progress

Mermaid diagram

Comments?

meta.yaml

Updates from DSOMM

Comments

10 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Improve data model and the loading service #380

Description

Data model

Drawbacks

Suggestion

1) Create a central load service

2) Split the Activity info and Team's progress

Mermaid diagram

Comments?

Activity

yashchauhan4579 commented on Mar 23, 2025

vbakke commented on Mar 27, 2025

wurstbrot commented on Mar 29, 2025

vbakke commented on Mar 30, 2025

meta.yaml

Updates from DSOMM

Comments

vbakke commented on Apr 8, 2025

wurstbrot commented on Apr 10, 2025

vbakke commented on Apr 17, 2025

vbakke commented on Apr 18, 2025

vbakke commented on May 4, 2025

vbakke commented on May 4, 2025

wurstbrot commented on May 4, 2025

vbakke commented on May 4, 2025

10 remaining items

vbakke commented on Jul 13, 2025

vbakke commented on Aug 2, 2025

vbakke commented on Aug 5, 2025

vbakke commented on Aug 17, 2025

vbakke commented on Aug 31, 2025

vbakke commented on Sep 13, 2025

vbakke commented on Sep 21, 2025

vbakke commented on Oct 19, 2025

github-actions commented on Nov 24, 2025

vbakke commented on Nov 28, 2025

vbakke commented on Dec 21, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions