1. CVSS (Common Vulnerability Scoring System)
- What it is: A standardized framework for rating the severity of software vulnerabilities. Widely used across the security industry.
- How it's Calculated:
- Based on a formula considering multiple base metrics capturing the vulnerability's inherent characteristics: Attack Vector, Attack Complexity, Privileges Required, User Interaction, Scope, Confidentiality Impact, Integrity Impact, Availability Impact.
- Each metric has defined values (e.g., Attack Vector: Network, Adjacent, Local, Physical).
- Outputs a numerical score from 0.0 to 10.0, often mapped to qualitative ratings (Low, Medium, High, Critical).
- Also includes optional Temporal (exploit maturity, remediation level) and Environmental (specific organizational context) metrics that can modify the score, but the Base score is the universal standard part.
- Learnings for Instrumentation Score:
- Standardization is Key: CVSS's strength is its defined, public, vendor-neutral set of metrics and formula. The Instrumentation Score needs this level of specification clarity to be adopted.
- Multi-dimensional: Quality/Risk is rarely a single thing. CVSS breaks it down into understandable components. Instrumentation Score's rule-based approach with impact levels aligns well here.
- Transparency: The calculation method is open, allowing anyone to understand why a vulnerability gets a specific score. Instrumentation Score needs this transparency.
- Context vs. Standard: CVSS separates the universal Base score from contextual modifiers (Temporal/Environmental). The Instrumentation Score should likely focus first on a standardized "Base Instrumentation Score".
2. Google Lighthouse / PageSpeed Insights
- What it is: An open-source, automated tool for improving the quality of web pages. Provides scores for Performance, Accessibility, Best Practices, and SEO.
- How it's Calculated:
- Runs a series of audits against a page.
- The Performance score (0-100) is a weighted average of specific metrics (Largest Contentful Paint, Total Blocking Time, Cumulative Layout Shift, etc.). Weights are periodically adjusted based on user experience research.
- Other scores (Accessibility, Best Practices, SEO) are often based on passing/failing a set of specific checks relevant to that category.
- Crucially, provides actionable opportunities linked directly to failed audits or poor metrics.
- Uses real-world performance data (from HTTP Archive) to define the score distribution curves (log-normal), meaning scores reflect performance relative to other sites.
- Learnings for Instrumentation Score:
- Actionability is Paramount: Lighthouse excels because it doesn't just give a score; it tells you what to fix and why. The Instrumentation Score must be tightly integrated with actionable recommendations based on the rules that were triggered.
- Weighted Averages: Combining multiple factors with different levels of importance (weights) is a proven model for a composite score. Instrumentation Score's impact levels serve a similar purpose.
- Clear Categories: Separating scores into distinct categories (Performance, Accessibility, etc.) helps users focus. While Instrumentation Score might start with one overall score, thinking about sub-scores (e.g., Resource Attribute Quality, Span Completeness Quality) could be a future evolution.
- Data-Driven Thresholds: Using real-world data to set score thresholds makes them more meaningful. This is harder initially but a long-term goal could be to base Instrumentation Score thresholds on observed data patterns.
3. SonarQube Quality Gate
- What it is: A tool for continuous inspection of code quality, providing metrics and enforcing standards. The "Quality Gate" is a key concept.
- How it's Calculated:
- Doesn't typically provide a single numerical score out-of-the-box, but rather a Pass/Fail status on a configurable "Quality Gate".
- The Quality Gate is defined by a set of conditions based on various code metrics (e.g., Code Coverage > 80%, Reliability Rating = A, No new Blocker bugs, Duplication % < 3%).
- Conditions often focus specifically on new code ("Clean as You Code" approach).
- Underlying metrics include things like complexity, duplications, bugs/vulnerabilities/smells (often rated A-E), test coverage (%), etc.
- Learnings for Instrumentation Score:
- Focus on "New Code": The idea of focusing rules on changes or recent activity can be powerful for driving incremental improvement, though maybe complex for an initial score. Instrumentation Score's time-bound score (30 days) captures some of this.
- Pass/Fail Thresholds: While Instrumentation Score aims for a numerical score, having defined thresholds that trigger a "Needs Improvement" or "Poor" status (like a failed Quality Gate) can simplify interpretation and decision-making.
- Driving Developer Behavior: Quality Gates are often integrated into CI/CD pipelines to block merges/deployments, directly influencing developer workflows. The Instrumentation Score could eventually serve a similar purpose in observability reviews or pipeline gates.
- Configurability vs. Standard: SonarQube allows customizing gates. The Instrumentation Score standard should likely be less configurable initially to be a standard, but understanding this trade-off is important.
4. SSL Labs Server Test
- What it is: A widely respected free online service that performs a deep analysis of the configuration of any SSL/TLS web server.
- How it's Calculated:
- Performs numerous specific checks across categories: Certificate validity/trust, Protocol Support, Key Exchange strength, Cipher Strength.
- Assigns scores (0-100) to each category based on best practices (e.g., TLS 1.3 support = 100%, TLS 1.2 = 90%, weak ciphers penalized).
- Combines category scores into an overall numerical score.
- Translates the numerical score into a letter grade (F to A+).
- Applies specific rules that can cap or boost the grade (e.g., no TLS 1.3 support caps grade at A-; HSTS required for A+; known vulnerabilities result in F).
- Learnings for Instrumentation Score:
- Rewarding Excellence (A+): Having a top tier that requires specific best practices encourages going beyond "good enough". Instrumentation Score's idea of potential bonus points aligns.
- Grade Caps for Deficiencies: Critical issues directly limit the maximum achievable score, regardless of other positive factors. Instrumentation Score does not cap the score; it gives Critical and Important rules higher weights instead.
- Clear Criteria: SSL Labs is trusted because its criteria (while complex) are documented and based on established security best practices. The Instrumentation Score rules need to be grounded in OTel specs and community consensus.
- Evolution: SSL Labs grading evolves as best practices change (e.g., requirements for TLS versions, key lengths). The Instrumentation Score governance needs to allow for similar evolution.
Overall Lessons:
- Transparency is Non-Negotiable: How the score is calculated must be public and clear.
- Actionability Drives Value: The score needs to directly guide users on how to improve.
- Multiple Factors Matter: Combine different aspects of quality, potentially with weighting or criticality, for a meaningful assessment.
- Clear Scope and Purpose: Define exactly what the score measures and what it doesn't.
- Standardization Enables Comparison: A common definition allows for benchmarking and shared understanding.
- Governance is Required: A process for maintaining and evolving the standard is essential for long-term relevance.