AI Coding Tools -- Dashboard Pro

TL;DROne dashboard for Claude Code, Codex and Kiro usage and spend, deployed per client on the client's own Grafana Cloud stack and powered by AWS; live for BFJFunds and for Jedi Hill.Claude Code and Codex send OpenTelemetry straight to Grafana Cloud; Kiro's daily reports and AWS bill arrive through Amazon Simple Storage Service (Amazon S3), AWS Lambda and AWS Cost Explorer forwarders.The AI feature is the monthly AI Adoption Brief: an AWS Step Functions workflow in which code computes every figure, Amazon Bedrock (Claude) writes the brief, and Amazon Bedrock Guardrails plus a number check run before it reaches leadership.Value: conclusions instead of charts, idle paid seats with their monthly cost, and one view of AI spend across tools. Metrics cover before and after both the dashboard and the brief.Started 2026-07-15, ongoing.

1. Overview

What it is. A unified usage dashboard for AI coding tools (Claude Code, Codex and Kiro) on the client's own Grafana Cloud stack, and the AI Adoption Brief: a monthly report for leadership, written on Amazon Bedrock from figures computed in code, that says what changed in AI adoption and spend, why, and what to do about it.

The problem it solves. Engineering organisations now pay for several AI coding tools at once. Usage and cost sit in separate consoles with different units and different gaps, so nobody can answer "how much do we spend on AI, who uses it, and is adoption growing?" without collecting the data by hand. Even with charts, a leader who has to justify the spend each month still works out the story alone: which paid seats are idle, what an active user costs, whether one tool is replacing another.

Who uses it. Engineering leadership (CTO, engineering managers) and the platform or finance owners of the AI budget. Live for BFJFunds (sigmajedi.grafana.net, single sign-on) and for Jedi Hill's own team.

2. Timeline

Item	Value
Start	2026-07-15: Grafana monitoring dashboard and company-wide token usage reporting, raised as action A22 in the weekly meeting
End	Ongoing

Date	Milestone
2026-07-15	Grafana dashboard for AI token usage and cost raised (weekly meeting, A22)
2026-07-16	BFJFunds's Q3 plan includes unified AI and LLM management, with usage and efficiency monitoring
2026-07-24	Managed OpenTelemetry settings for Claude Code and Codex defined
2026-07-27	Statement of work: unified Claude Code and Codex dashboard on BFJFunds's Grafana stack
2026-08-02	First combined dashboard, telemetry validated on a test VM
2026-08-05	Dashboard parameterised and deployed to BFJFunds's tenant
2026-08-12	BFJFunds dashboard review and trial feedback
2026-09-19	Kiro's native OTLP export found to be rejected by Grafana Cloud; Amazon S3 report path designed instead
2026-09-27	Kiro added to the Jedi Hill dashboard
2026-10-13	Kiro forwarder made deployable per client, with Kiro billing from AWS Cost Explorer
2026-10-14	Client onboarding packaged as a Claude Code agent skill
2026-10-16	AI Adoption Brief design agreed
2026-10-20	Kiro added to the BFJFunds dashboard

3. Architecture

Everything is deployed per client, on AWS using the client's own account and Grafana Cloud stack, with Terraform and dashboard-as-code. No client data passes through Jedi Hill's systems.

3.1 Data collection

Tool	Path	Freshness
Claude Code (including Claude Code inside Claude Desktop)	OpenTelemetry metrics and events sent straight to Grafana Cloud (Prometheus and Loki). Settings are distributed through Claude's managed settings; user.email is attached by the subscription	Near real time
Codex	OpenTelemetry events to Grafana Cloud Loki; settings per host	Near real time
Kiro	Kiro Enterprise writes per-user daily reports to an encrypted Amazon S3 bucket in the client account. An Amazon S3 event triggers kiro-report-forwarder (AWS Lambda), which joins the two reports by user, and pushes metrics to Prometheus and one summary line per user, client and model to Loki	One day behind (report at 02:00 UTC)
Kiro billing	A daily Amazon EventBridge rule runs kiro-billing-forwarder (AWS Lambda), which reads Kiro's actual AWS charges (seats and overage) from AWS Cost Explorer, re-sending the last three days as AWS Cost Explorer revises them	Daily

Kiro's own OTLP export is left enabled but cannot be ingested: Kiro stamps data at the end of the activity day and delivers it two hours later, outside Grafana Cloud's out-of-order window. The Amazon S3 path stamps data at ingestion time instead.

3.2 Dashboard

• One dashboard per client ("AI Coding Tools — Combined Usage") with sections for cost and spend, adoption and activity, engineering productivity, and per-tool diagnostics; filters for tool and user.

• Generated as code (gen_dashboard.py) and deployed with gcx, Grafana Cloud's resource-as-code CLI, so every client gets the same tested queries; only the user roster and the enabled tools vary.

• Data gaps are stated on the panels, not hidden: Codex reports no cost, lines of code, commits or pull requests; Kiro reports credits rather than tokens and its cost is account-level only.

3.3 AI Adoption Brief

Step	Service	Does
Schedule	Amazon EventBridge Scheduler	Starts the workflow on the 1st of each month
Orchestrate	AWS Step Functions	Runs the steps below, with retries and failure alarms
Collect	AWS Lambda	Queries Grafana Cloud with a read-only token from AWS Secrets Manager; reads the client's seat table (tool, seats, unit price) from Amazon S3
Compute	AWS Lambda	Computes every figure in plain code
Narrate	Amazon Bedrock, Claude through the Converse API	Turns the computed facts into the brief
Guard	Amazon Bedrock Guardrails and a check AWS Lambda function	Masks personal data and verifies that every number in the text matches a computed fact
Deliver	AWS Lambda, Amazon S3, Amazon Simple Email Service (Amazon SES)	Archives the brief in Amazon S3, emails it, posts it to Lark or Slack, and links it from a dashboard panel

3.4 Security and operations

• Least-privilege roles per AWS Lambda function; the Grafana token is read-only and stored in AWS Secrets Manager; Amazon S3 buckets are encrypted with AWS Key Management Service (AWS KMS).

• The client's Grafana stack is accessed through the client's SSO; dashboard folder permissions are set on every deploy.

• The Kiro forwarder de-duplicates re-sent reports; a verification script checks that a given day's data arrived.

• A Claude Code agent skill guides onboarding a new client, adding a tool and upgrading an existing client, and diffs the live dashboard against the repository.

4. AI feature design

4.1 What the brief contains

Section	Content
Adoption	Monthly active users per tool, change from the previous month, new and dropped users against the roster
Spend and efficiency	Cost per active user and per tool, on the dashboard's cost basis: Claude Code's reported cost plus Kiro's AWS bill; Codex reports no cost
Seat utilisation	Paid seats from the client's seat table against active users, and the monthly cost of idle seats
Depth of use	Sessions, tool-call acceptance, model mix, power users and occasional users
Anomalies with likely causes	For example, Kiro usage falling in the same week Claude Code rises
Recommendations	Seat right-sizing, enablement targets, tool consolidation
Data caveats	Kiro's one-day lag, Codex's missing cost data, and any day of missing data

4.2 Design principle: the model explains, code calculates

Every number comes from code; the model only writes. Before sending, an automated check compares every number in the text with the computed facts, and Amazon Bedrock Guardrails filter the output. This makes the brief accurate by construction and testable.

4.3 Responsible AI controls

Area	Control
Accuracy	Facts computed in code; numbers checked against them before delivery; a brief that fails the check is not sent
Privacy	Organisation-level figures by default; naming individuals is a per-client setting; personal data masked by Amazon Bedrock Guardrails
Data residency	Powered by AWS and running on the client's own account; data moves only between the client's Grafana stack and the client's account
Human oversight	Optional draft, approve, send step for a client's first months
Evaluation	A golden set of past months with expected findings, re-run whenever the prompt or model changes
Cost	One Amazon Bedrock call per client per month; Amazon CloudWatch alarms on failures

4.4 Value

• From charts to conclusions. Leaders get the month's story and recommended actions instead of reading panels for three tools with different gaps.

• Direct savings. Idle paid seats are listed with their monthly cost, the most concrete dollar value of the brief.

• One view of AI spend. Claude Code, Codex and Kiro in one place, per user and per tool, with each tool's data limits stated.

• Repeatable delivery. Per-client Terraform, dashboard-as-code and an onboarding agent skill make a new client a configuration change.

5. Metrics

Baselines are measured at onboarding, before the dashboard or the brief is in use; actuals after. The same definitions serve as the project's ongoing benchmark.

5.1 Before and after the dashboard

#	Metric	Definition and formula	Data source	Baseline	Target	Actual
D1	Time to answer the spend question	Hours to produce total monthly AI spend and active users across all tools	Before: client interview or timed exercise. After: dashboard (minutes)	TBD	TBD	TBD
D2	Coverage	AI coding tools with usage visible in one place ÷ AI coding tools the client pays for	Dashboard, client tool list	TBD	100%	TBD
D3	Seat utilisation	Active users in the month ÷ paid seats, per tool	Dashboard Active Users, client seat table	TBD	TBD	TBD
D4	Cost per active user	Monthly cost ÷ active users, on the dashboard's cost basis	Dashboard Total Cost and Active Users	TBD	TBD	TBD
D5	Adoption	Monthly active users per tool and month-on-month change	Dashboard	TBD	TBD	TBD
D6	Data freshness	Delay from use to visibility, per tool	Telemetry timestamps	n/a	Near real time; Kiro one day	TBD

5.2 Before and after the AI Adoption Brief

#	Metric	Definition and formula	Data source	Baseline	Target	Actual
B1	Analysis effort	Person-hours per month to produce the adoption and spend analysis	Before: client estimate. After: time to read and approve the brief	TBD	TBD	TBD
B2	Time to insight	Days from month end to the analysis reaching leadership	Before: client process. After: delivery timestamp	TBD	Day 1 of the month	TBD
B3	Idle seat cost identified	Σ over tools of (paid seats − active users) × seat price, per month	Brief, seat table	n/a	TBD	TBD
B4	Savings realised	Monthly cost of seats removed or downgraded after a brief recommended it	Client seat table changes, AWS bill	n/a	TBD	TBD
B5	Numeric accuracy	Numbers in the brief matching a computed fact ÷ numbers in the brief	Guard step results	n/a	100%	TBD
B6	Evaluation score	Expected findings found ÷ expected findings, on the golden set	Evaluation run	n/a	TBD	TBD
B7	Run cost	Amazon Bedrock and AWS cost per brief	AWS Cost Explorer	n/a	TBD	TBD