DevSecTrust: Standardising How We Measure Software Development Security

Metric-based software security plays a crucial role in allowing software developers to make informed decisions about their development practices, while also allowing software users to evaluate the security risks associated with the software they use. Metrics are increasingly used to ensure code security, but there has been little formal evaluation of their broader applicability to date, and interpretation of their results remains a qualitative task. To address this gap, we introduce DevSecTrust, a standardised evaluation framework for measuring and comparing software development security metrics. DevSecTrust provides: (i) a unified control-mapped metric schema, (ii) outcome-based calibration and validation against real vulnerability and maintenance data, and (iii) robustness and manipulability testing to assess metric reliability. This paper analyses two software development security tools, MITRE’s Hipcheck and OpenSSF’s Scorecard, to evaluate and contrast the metrics they produce against widely used open-source software projects. Our quantitative comparison identified low correlation and inconsistent distributions between the tools’ outputs, and our qualitative analysis of feature weighting and scoring logic revealed foundational differences in how each tool conceptualises “secure development”. These inconsistencies complicate trust in development security metrics and hinder their interpretability and operational value. This contributes to a path toward standardised measurement of software development security.