I looked at the top 200 Java projects, out of curiosity, to see if any of the projects I'm working on, like GeoTools, is included in the list. It was not, which is not an issue per se, but then I've computed the criticality score from command line, getting this:

criticality_score --repo "https://github.com/geotools/geotools"
name: geotools
url: https://github.com/geotools/geotools
language: Java
created_since: 111
updated_since: 0
contributor_count: 315
org_count: 6
commit_frequency: 9.7
recent_releases_count: 16
closed_issues_count: 150
updated_issues_count: 161
comment_frequency: 1.0
dependents_count: 337
criticality_score: 0.66477

The score alone would place the project at around position 100 of the top 200 projects. Since it's a no show, I'm wondering if there is any other criteria used to include/exclude projects, besides the pure score?

For many projects the github creation date might not match the project creation date.

Would it be better to look at the date of the oldest commit in the repository?

For example, for OpenSSL the computed creation_since value is 95 months, as the date of creation of a github mirror (2013-01-15T22:34:48Z), but the project is almost 22 years old (the first commit in the master branch dates back to 1998-12-21T10:52:45+00:00)!

The cap for the field is 10 years anyway, so it's not that bad, but still it is one parameter in the equation that might be adjusted.

Edit: this also affects other fields (e.g. recent_releases) when they are computed based on estimates based on the time since creation.

Thoughts?

I asked for the criticality info on several projects in my industry's ecosystem, and the dependents_count really confuses me and makes me suspicious about how it's computed. Some of the projects I checked are hard dependencies of others, so if transitive dependencies are being properly tracked, the former should always have higher dependents_count than the latter, no? But this is not the case.

One project that I run is very specialized and is of no use to casual small projects, only making sense as an embedded component of a large open source or commercial app. So while certainly very important in my industry and having a large number of end users touch those things in which it is embedded, I expect it to have a tiny number of directly downstream projects. Yet it has an absurdly, implausibly high dependents_count. Other projects I checked on that I know are directly used by orders of magnitude more projects, have implausibly low dependents_count.

Is there some kind of verbose mode that prints details that would give us more information about how these scores are computed? Like, more insight into why it thinks a project has few or many dependent projects?

I should mention that these are C++ projects, so perhaps the means by which dependencies are tracked is very flawed compared to a python (say) which may have a requirements.txt. How is it computed for C++? Has anybody considered promoting a GitHub convention of having a particularly named file serve as a manifest for what other projects a code base is dependent on? (Informationally only, since no C++ build system cares about such things.)

Hi,

I just saw that the Maven and Gradle projects are less important that 2000 java projects where they are used in as a build tool. Maybe due to the fact that they:

• are not a declared dependency
• https://github.com/ossf/criticality_score/issues/14
• https://github.com/ossf/criticality_score/issues/23
• external issue tracker
• All the parts (pluggable, not a dependency!) are split into many repositories
• Mosten downloaded via maven.org, sdkman, package systems, etc

Probably the same for other languages and build-tools, but haven’t checked.

I get:

$ pip3 install criticality-score
Collecting criticality-score
  Could not find a version that satisfies the requirement criticality-score (from versions: )
No matching distribution found for criticality-score

I wanted to submit a suggestion to include GitHub Watchers (to help assess popularity) and the GitHub Description (to clarify the project's overall goal). I am currently helping contribute to OSSF's Security Metrics project, in which we are retrieving several of the GitHub metrics covered in this project (but also need to analyze the two mentioned above to help with our overall security assessment). If these can be included via the pull request I have submitted that would be extremely helpful. Thank you!

When I was running the script, I bumped into these repos that they fall into the filter due to high number of stars but they're actually empty and the script throws an exception: https://github.com/fossasia/libregraphics.asia https://github.com/libredesktop/libredesktop-events https://github.com/libredesktop/libredesktop-project-list https://github.com/libredesktop/LibreDesktop-Specs https://github.com/meilix/arch-meilix https://github.com/meilix/deb-meilix https://github.com/meilix/meilix-addons https://github.com/meilix/meilix-art https://github.com/meilix/meilix-connect https://github.com/meilix/meilix-web https://github.com/susiai/susi_partners https://github.com/susiai/susi_sdk https://github.com/ascoders/blog https://github.com/bigdongdongCLUB/newGCP https://github.com/koush/support-wiki https://github.com/mariobehling/ai-packages https://github.com/mariobehling/mb-sandbox https://github.com/meilix/meilix-docs https://github.com/paulirish/devtools-addons https://github.com/QingDaoIT/BlackList https://github.com/zhengzhouqiuzhi/zhengzhouqiuzhi

To handle it, for GitLab, checking the commits length was enough:

if len(repo.commits.list()) == 0:

For GitHub, I couldn't find any proper way to understand whether the repo is empty. When we call "get_commits().totalCount", it already throws an exception. What I did is to force it to throw the exception by assigning "totalCount" to an unused variable (I could do it by printing the value as well?). Not an ideal solution, so let me know what you think.

try:
	repo = get_github_auth_token().get_repo(repo_url)
	# Validate whether repo is empty; if it's empty, calling totalCount throws a 409 exception
	total_commits = repo.get_commits().totalCount
except github.GithubException as exp:
	if exp.status == 404 or exp.status == 409:
		return None
return GitHubRepository(repo)

Another remark is that we're spending one more request from our rate limit when calling "get_commits()" to make this validation. I only tested this for GitHub, but I'm assuming it's the same for GitLab as well.

Alternatively, we can make all these calls before initializing the repo, do the validations, and pass them to repo object as arguments? This would also help us reducing the number of call to the API, but making these changes would take some time.

To be able to test my changes, I created empty repos on both GitHub & GitLab btw: https://github.com/coni2k/empty-repo https://gitlab.com/coni2k/empty-repo

Last, I also added this bit to "generate" script. Otherwise it fails when there are no processed repos:

if len(stats) == 0:
    return

The new --repolist parameter takes the name of a file containing a list of repositories to score.

usage: run.py [-h] (--repo REPO | --repolist REPOLIST | --local-file L_FILE) [--format {default,csv,json}] [--params PARAMS [PARAMS ...]]

Gives criticality score for an open source project or a list of projects.

optional arguments: -h, --help show this help message and exit --repo REPO repository url --repolist REPOLIST listfile of repository urls --local-file L_FILE path of a local csv file with repo stats --format {default,csv,json} output format. allowed values are [default, csv, json] --params PARAMS [PARAMS ...] Additional parameters in form ::<max_threshold>

This at least partially addresses Issue #97

Signed-off-by: Arnaud J Le Hors [email protected]

Spark has much higher score than ElasticSearch and Beam, Spark is missing but ElasticSearch and Beam are there, why?

apache/spark:

$ criticality_score --repo github.com/apache/spark
name: spark
url: https://github.com/apache/spark
language: Scala
created_since: 83
updated_since: 0
contributor_count: 2374
org_count: 4
commit_frequency: 53.8
recent_releases_count: 20
closed_issues_count: 1252
updated_issues_count: 1456
comment_frequency: 12.1
dependents_count: 396346
criticality_score: 0.96476

elastic/elasticsearch:

$ criticality_score --repo github.com/elastic/elasticsearch
name: elasticsearch
url: https://github.com/elastic/elasticsearch
language: Java
created_since: 132
updated_since: 0
contributor_count: 1709
org_count: 3
commit_frequency: 127.1
recent_releases_count: 21
closed_issues_count: 7966
updated_issues_count: 9234
comment_frequency: 1.0
dependents_count: 95320
criticality_score: 0.88175

apache/beam:

$ criticality_score --repo github.com/apache/beam
name: beam
url: https://github.com/apache/beam
language: Java
created_since: 59
updated_since: 0
contributor_count: 980
org_count: 7
commit_frequency: 67.1
recent_releases_count: 7
closed_issues_count: 725
updated_issues_count: 826
comment_frequency: 4.3
dependents_count: 11397
criticality_score: 0.8319

I am directly responsible for two open source projects. I was shocked to see that one is on your "top 200" list of C++ projects. The other project has been around longer, has more contributors, more PRs, surely has an order of magnitude more downstream users, and in fact has a much higher criticality score. But it's not on the list. I can't quite figure out what the top 200 would be measuring (I would think the 200 projects with the very highest criticality score itself? But apparently not?) for the first project to show up on the list but not the other.

Can you give any insight about WHAT is being ranked in your "top" lists?

tsdx, a TypeScript project generator, appears in the top 200 list for JavaScript packages; however, TypeScript itself does not. That seems somewhat counterintuitive.

Bumps cloud.google.com/go/bigquery from 1.43.0 to 1.45.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps actions/checkout from 3.1.0 to 3.3.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps google.golang.org/api from 0.102.0 to 0.106.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

• Fixed issue https://deps.dev/advisory/osv/GO-2022-1144

https://deps.dev/go/github.com%2Fossf%2Fcriticality_score

Signed-off-by: nathannaveen [email protected]

• Published the docker images to ghcr.io
• Here is an example: https://github.com/nathannaveen?tab=packages&repo_name=criticality_score, https://github.com/nathannaveen/criticality_score/actions/runs/3810049085
• After https://github.com/ossf/criticality_score/pull/293 gets merged in, I will include docker images for scorer.

Signed-off-by: nathannaveen [email protected]