Privacy Policy

About CLIWatch

CLIWatch is a CLI eval platform built and maintained as part of the agentic-federation GitHub organization. This policy explains what data we collect, how we use it, and your rights.

Data Controller

CLIWatch is operated by its parent company in the Netherlands. The operating company is the data controller for the personal data processed through this service, as defined under the EU General Data Protection Regulation (GDPR).

What We Collect

Account information: your name and email address, provided via OAuth (GitHub or Google).

Workspace data: workspace names, member lists, and roles.

Benchmark reports: CLI names, pass rates, model results, task definitions, task traces, and token usage data that you upload.

Usage logs: API request metadata (timestamps, endpoints, response codes) for operational monitoring.

How We Use It

We use your data to provide the CLIWatch service: storing and displaying your eval results, powering the public leaderboard (for opted-in CLIs), processing billing, and improving the product. We do not sell your data.

Legal Basis for Processing

Under the GDPR, we process your personal data on the following legal bases:

• Contractual necessity (Article 6(1)(b)): processing required to provide the CLIWatch service, manage your account, and fulfill our agreement with you
• Legitimate interest (Article 6(1)(f)): product improvement, operational monitoring, and fraud prevention, where these interests are not overridden by your rights
• Consent (Article 6(1)(a)): displaying your CLI on the public leaderboard, which requires your explicit opt-in. You can withdraw consent at any time

Third Parties

We share data with the following third-party services, only as needed to operate CLIWatch:

• AWS for cloud infrastructure and data storage
• Stripe for payment processing
• Vercel for static site hosting
• WorkOS for authentication and identity management
• OAuth providers (GitHub, Google) for social login via WorkOS
• LLM providers (Anthropic, OpenAI, Google) via the Vercel AI Gateway for running eval agents. The AI Gateway acts as a proxy that routes requests to the appropriate LLM provider. During eval execution, task definitions and CLI output are sent through the gateway to these providers. We do not send your account information or billing data to LLM providers.

AI Agents & Eval Data

CLIWatch evals run in your own environment. You install the @cliwatch/cli-bench npm package and execute evals in your own CI (e.g. GitHub Actions) or locally on your machine. During an eval run, the tool gives LLM agents access to your CLI and sends task definitions and CLI output to third-party LLM providers via the Vercel AI Gateway. Results are then uploaded to the CLIWatch API.

Because evals run in your environment, you control what data is accessible. We recommend the following precautions:

• Run evals in CI or an isolated environment rather than on a machine with access to sensitive data
• Only the task definition and CLI output required for the eval are sent to LLM providers
• Do not include secrets, credentials, or sensitive personal data in task definitions or CLI configurations used for evals
• Agent outputs are stored as eval traces in your workspace and are not shared unless you opt in to the public showcase

AI agent behavior is inherently non-deterministic. We cannot guarantee that an agent will never produce unexpected output or interact with a CLI in an unintended way.

Cookies & Local Storage

We use cookies and local storage for session authentication and theme preferences.

International Data Transfers

CLIWatch is operated from the Netherlands. Your data may be transferred to and processed in countries outside the European Economic Area (EEA), including the United States, where our infrastructure providers (AWS, Vercel) and LLM providers (Anthropic, OpenAI, Google) operate. Where data is transferred outside the EEA, we rely on Standard Contractual Clauses (SCCs) approved by the European Commission or adequacy decisions to ensure an appropriate level of data protection.

Data Retention

Your data is kept for as long as your account is active. If you delete your account, we will delete your data. You can request data deletion at any time through our contact page.

Your Rights

Under the GDPR, you have the right to:

• Access your personal data and receive a copy
• Rectify inaccurate or incomplete data
• Erase your personal data (“right to be forgotten”)
• Restrict processing of your data
• Port your data to another service. You can export eval data at any time via the API.
• Object to processing based on legitimate interest
• Withdraw consent at any time, where processing is based on consent

To exercise these rights, reach out via our contact page. You also have the right to lodge a complaint with the Dutch Data Protection Authority (Autoriteit Persoonsgegevens) at autoriteitpersoonsgegevens.nl.

Third-Party Links

CLIWatch may contain links to third-party websites and services (e.g. GitHub, Stripe customer portal, LLM provider documentation). We are not responsible for the privacy practices or content of these external sites. We encourage you to review their privacy policies before providing them with your data.

Security

All data is encrypted in transit via TLS. API keys are hashed before storage. Access to production systems is restricted to authorized personnel with role-based access controls.

Children

CLIWatch is not designed for or directed at anyone under the age of 16. We do not knowingly collect data from children.

Changes

We may update this policy from time to time. When we do, we will update the “last updated” date at the top and notify you via email or in-app notification.

Contact

Questions about this policy? Reach out via our contact page.