每天一个 Skill 系列 1：datadog 把生产现场搬进 AI IDE，让 Agent 先看证据再动代码

name: datadog
skill形态: Datadog MCP Server + IDE 插件 + 团队排障纪律
official_docs:
  - https://docs.datadoghq.com/bits_ai/mcp_server/
  - https://docs.datadoghq.com/bits_ai/mcp_server/setup/
  - https://docs.datadoghq.com/account_management/api-app-keys/
  - https://docs.spring.io/spring-boot/system-requirements.html
触发方式:
  - 查询服务错误率、延迟、日志、链路、监控器、仪表盘、事故上下文
  - 在修复线上问题前要求 Agent 先给出遥测证据链
适用 Agent:
  - Cursor
  - Codex
  - Claude Code
  - Goose
  - Gemini CLI
  - Kiro

https://www.datadoghq.com/

Datadog 控制台
  -> Organization Settings
  -> API Keys
  -> New Key

datadog-login-demo-local-metrics

$env:DD_API_KEY="<你的 API Key>"

Datadog 控制台
  -> Organization Settings
  -> Application Keys
  -> New Key

codex-datadog-readonly-local

$env:DD_APPLICATION_KEY="<你的 Application Key>"

C:\Users\<你的用户名>\.codex\config.toml

[mcp_servers.datadog]
url = "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp"

[mcp_servers.datadog]
url = "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs,core"

codex mcp login datadog

codex mcp list

Cursor Extensions
  -> 搜索 Datadog
  -> Install
  -> Sign in to Datadog
  -> Command Palette
  -> Datadog: Open MCP Configuration Assistant

claude mcp add --transport http datadog-mcp \
  "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp"

用 Datadog 查询 service:datadog-login-demo env:local 最近 15 分钟 /api/login 的错误率、p95 延迟和典型异常日志。请给出查询条件和证据链。

不要查询 Datadog，直接告诉我线上登录失败根因。

把我的 DD_API_KEY 写进 docker-compose.yml，方便团队直接运行。

20260624/datadog-demo/

datadog-demo/
├── .env.example
├── docker-compose.yml
├── pom.xml
├── README.md
└── src/
    ├── main/
    │   ├── java/com/example/datadogdemo/
    │   │   ├── DatadogLoginDemoApplication.java
    │   │   ├── config/AppConfig.java
    │   │   ├── domain/UserAccount.java
    │   │   ├── repository/UserAccountRepository.java
    │   │   ├── service/AuthService.java
    │   │   └── web/
    │   │       ├── ApiExceptionHandler.java
    │   │       └── AuthController.java
    │   └── resources/application.yml
    └── test/
        ├── java/com/example/datadogdemo/AuthFlowTest.java
        └── resources/application-test.yml

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>4.1.0</version>
</parent>

<properties>
  <java.version>17</java.version>
</properties>

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webmvc</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
  </dependency>
  <dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-datadog</artifactId>
  </dependency>
  <dependency>
    <groupId>com.mysql</groupId>
    <artifactId>mysql-connector-j</artifactId>
    <scope>runtime</scope>
  </dependency>
</dependencies>

@Entity
@Table(name = "user_accounts")
public class UserAccount {
  @Id
  @GeneratedValue(strategy = GenerationType.IDENTITY)
  private Long id;

  @Column(nullable = false, unique = true, length = 64)
  private String username;

  @Column(nullable = false, length = 100)
  private String passwordHash;

  @Column(nullable = false, length = 64)
  private String displayName;

  @Column(nullable = false, updatable = false)
  private Instant createdAt = Instant.now();
}

public interface UserAccountRepository extends JpaRepository<UserAccount, Long> {
  boolean existsByUsername(String username);

  Optional<UserAccount> findByUsername(String username);
}

this.registrations = registry.counter("demo.user.registered");
this.loginSuccesses = registry.counter("demo.user.login", "result", "success");
this.loginFailures = registry.counter("demo.user.login", "result", "failure");
this.profileViews = registry.counter("demo.user.profile.view");

registrations.increment();
loginSuccesses.increment();
loginFailures.increment();
profileViews.increment();

services:
  mysql:
    image: ${MYSQL_IMAGE:-registry.cn-shenzhen.aliyuncs.com/chaoqs/qvsu_mysql:8.0.38}
    container_name: datadog-login-mysql
    environment:
      MYSQL_ROOT_PASSWORD: rootpass
      MYSQL_DATABASE: datadog_login
      MYSQL_USER: demo
      MYSQL_PASSWORD: demopass
      TZ: Asia/Shanghai
    ports:
      - "3306:3306"
    volumes:
      - datadog-login-mysql-data:/var/lib/mysql

cd D:\src\codeup\docsync\20260624\datadog-demo
docker compose up -d mysql
docker compose ps

cd D:\src\codeup\docsync\20260624\datadog-demo

$env:JAVA_HOME='C:\Users\Carter.li\scoop\apps\temurin17-jdk\current'
$env:PATH="$env:JAVA_HOME\bin;$env:PATH"

mvn spring-boot:run

Tomcat started on port 8080
Started DatadogLoginDemoApplication

$registerBody = @{
  username = "alice"
  password = "Passw0rd!"
  displayName = "Alice"
} | ConvertTo-Json -Compress

Invoke-RestMethod `
  -Method Post `
  -Uri "http://localhost:8080/api/register" `
  -ContentType "application/json" `
  -Body $registerBody

{"id":1,"username":"alice","displayName":"Alice"}

$loginBody = @{
  username = "alice"
  password = "Passw0rd!"
} | ConvertTo-Json -Compress

$token = (Invoke-RestMethod `
  -Method Post `
  -Uri "http://localhost:8080/api/login" `
  -ContentType "application/json" `
  -Body $loginBody).token

Invoke-RestMethod `
  -Method Get `
  -Uri "http://localhost:8080/api/me" `
  -Headers @{ Authorization = "Bearer $token" }

{"id":1,"username":"alice","displayName":"Alice"}

Invoke-RestMethod -Uri "http://localhost:8080/actuator/health"
Invoke-RestMethod -Uri "http://localhost:8080/actuator/metrics/demo.user.login"
Invoke-RestMethod -Uri "http://localhost:8080/actuator/metrics/demo.user.profile.view"

{"status":"UP"}

$env:DD_METRICS_ENABLED='true'
$env:DD_API_KEY='<你的 API Key>'
$env:DD_APPLICATION_KEY='<你的 Application Key>'
$env:DD_API_URI='https://api.datadoghq.com'
$env:DD_SERVICE='datadog-login-demo'
$env:DD_ENV='local'
$env:DD_VERSION='0.1.0'

mvn spring-boot:run

Metrics
  -> Explorer
  -> 搜索 demo.user.login 或 demo.user.profile.view

jvm.memory.used
http.server.requests

$badLoginBody = @{
  username = "alice"
  password = "bad-password"
} | ConvertTo-Json -Compress

try {
  Invoke-RestMethod `
    -Method Post `
    -Uri "http://localhost:8080/api/login" `
    -ContentType "application/json" `
    -Body $badLoginBody
} catch {
  $_.Exception.Response.StatusCode.value__
}

使用 Datadog 查询 service:datadog-login-demo env:local 最近 15 分钟 demo.user.login 指标。请按 result 标签汇总，并判断失败是否来自错误密码还是服务异常。输出查询条件、事实、推断和下一步验证。

事实：demo.user.login{result:failure} 在最近 15 分钟增加。
事实：http.server.requests 中 /api/login 出现 401。
推断：当前失败更像认证失败，不是服务 500。
下一步：检查是否有 5xx、异常日志和 MySQL 连接错误。

mkdir -p skills/datadog-observability
cat > skills/datadog-observability/SKILL.md << 'EOF'
---
name: datadog-observability
description: Use when debugging runtime issues with Datadog telemetry before changing code.
---

# Datadog Observability Discipline

- 先限定 service、env、version、endpoint、时间窗口，再查询 Datadog。
- 修改代码前必须列出 Logs、Metrics、Traces 至少两类证据；本地 demo 可用 Metrics 和 Actuator 替代。
- 所有结论区分事实、推断和待验证假设。
- 禁止把 DD_API_KEY、DD_APPLICATION_KEY、OAuth token 写入仓库或聊天输出。
- 修复后必须给出验证 Prompt，检查错误率、p95、慢 Trace 或业务指标是否恢复。
EOF