TestPilot analyzes code changes and tells testers exactly what to test and why - saving 8-10 hours per week.
Critical gaps across all dimensions. Tool definitions exist but lack rigor. Descriptions are present but generic (avg ~80 chars, below 194-char baseline). Input schemas visible but incomplete: most parameters lack type constraints, enums, or validation rules. No output schemas documented. No error handling guidance. Security-critical tools (validateCredentials, processPayment, processRefund) expose sensitive operations without permission gates, confirmation patterns, or audit trail integration. Naming is inconsistent: 'validateCredentials' vs 'handleOTPLogin' mixes verb styles. No tool composition guidance. This server would not pass production code review.
Handles OTP-based login. Validates OTP expiry (5 minutes)
Multi-gateway support (Stripe, PayPal, Razorpay). Allows switching between payment providers
Logs security events to audit trail
Processes payment via payment gateway. Handles success/failure responses
Automated refund processing. Validates refund eligibility. Processes partial and full refunds. Sends confirmation email
Added XSS prevention. Escapes HTML special characters. Prevents script injection in profile fields
Sends password reset link to email. Link now expires in 30 minutes (was 1 hour)
initializePaymentGateway has empty input schema (no parameters defined). Tool is non-functional for agent composition.
Irreversible operations (processPayment, processRefund) lack confirmation patterns, dry-run support, and permission gates. Agents can trigger financial transactions without safeguards.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., does processPayment return transaction_id?). Breaks tool chaining.
Security-critical tools (validateCredentials, handleOTPLogin) lack audit trail integration and permission checks. No evidence of logSecurityEvent being called from auth flows.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 30 | 2026-07-28+ | v2 |
Updates user profile information. Validates email, phone, address
Age validation for GDPR compliance. Ensures users are 13+ (data protection laws). Blocks registration for underage users
Validates user email and password. Returns JWT token on success
Validates credit card or PayPal details
Validates phone number. Now supports international formats. Previously only validated US numbers (+1). Now supports: +44, +91, +61, +81, etc. Fixed regex to handle country codes properly
Validates reset token. Checks expiry and usage status. Now logs all reset attempts for security audit
Daily transaction limits for fraud prevention. Checks user tier (basic/premium/enterprise). Different limits per tier
Parameter descriptions lack format constraints, ranges, and validation rules. E.g., 'email' has no regex pattern, 'amount' has no min/max bounds, 'otp' has no length spec. Invites invalid LLM input.