Telemetry

Harpia includes a built-in Telemetry utility that acts as an “observability-as-a-service” tool directly within the framework. It tracks server traffic, page views, response times, errors, and visitor behavior out of the box, offering a powerful Reading API to build internal dashboards without needing third-party analytics tools.

Configuration

To use the telemetry module, instantiate it and place its handleRequest method as a global middleware.

import harpia, { Telemetry } from "@harpiats/core";

const app = harpia();

const telemetry = new Telemetry({
  ignore: ["/favicon.ico", "/healthcheck"], // Paths to exclude from monitoring
  trustProxy: true, // Enable if behind Nginx/Cloudflare to resolve real IPs
});

// Add as a global middleware
app.use(async (req, res, next) => {
  // Initialize the telemetry with the current request and IP
  await telemetry.initialize(req, app.requestIP() || "unknown");
  
  // Process the metrics
  const telemetryRes = await telemetry.handleRequest();
  
  // If the path is ignored, handleRequest might return a 204 Response early
  if (telemetryRes) return telemetryRes;
  
  next();
});

Storage

By default, the telemetry saves metrics in a memory store. If you are running multiple instances of your application or need persistent metrics across server restarts, you should pass a custom store (like RedisStore) to the constructor.

const telemetry = new Telemetry({
  store: new RedisStore(),
  ignore: ["/admin"]
});

Tracking Traffic Sources

telemetry.initialize() accepts a third argument of type TrafficSource, which lets you record the origin of each visit for marketing attribution.

app.use(async (req, res, next) => {
  const url = new URL(req.url);

  // Parse UTM parameters from the query string
  const trafficSource = {
    utm: {
      source: url.searchParams.get("utm_source") ?? undefined,
      medium: url.searchParams.get("utm_medium") ?? undefined,
    },
  };

  await telemetry.initialize(req, app.requestIP() || "unknown", trafficSource);

  const telemetryRes = await telemetry.handleRequest();
  if (telemetryRes) return telemetryRes;

  next();
});

Security Options

Because telemetry data can contain sensitive business intelligence and IP addresses, Harpia provides native security primitives to protect your Reading API. You do not need external dependencies or complex middlewares.

Configure accessToken or allowedIps (or both) during initialization:

const telemetry = new Telemetry({
  accessToken: process.env.TELEMETRY_TOKEN, // Requires this token on read methods
  allowedIps: ["127.0.0.1", "10.0.0.5"],    // Only allows read access from these IPs
});

When calling any method from the Reading API, simply pass the token and the caller’s IP inside the parameter object. If the validation fails, the method will throw an Unauthorized error.

Reading API (Queries)

Harpia’s Telemetry provides a comprehensive set of asynchronous query methods to retrieve and analyze your data.

getAll()

Retrieves the complete, raw telemetry data object.

const data = await telemetry.getAll({ token, callerIp });

// Example Response
{
  "access": {
    "totalRequests": 1500,
    "visitorsByDate": {
      "2023-10-25": {
        "192.168.1.1": {
          "totalRequests": 5,
          "visits": [
            { "path": "/home", "timestamp": "2023-10-25T14:00:00.000Z", "responseTime": 45, "error": null },
            { "path": "/about", "timestamp": "2023-10-25T14:01:00.000Z", "responseTime": 30, "error": null }
          ]
        }
      }
    }
  },
  "behavior": {
    "pageViews": {
      "/home": 850,
      "/about": 320
    }
  }
}

summary()

Returns a high-level overview of the day’s metrics, aggregating totals, unique visitors, average response time, and the top pages.

const stats = await telemetry.summary({ date, limit: 5, token, callerIp });

// Example Response
{
  "date": "2023-10-25",
  "totalRequests": 1500,
  "uniqueVisitors": 320,
  "topPages": [
    { "path": "/home", "views": 850 },
    { "path": "/about", "views": 320 }
  ],
  "avgResponseTime": 42.5,
  "totalErrors": 2
}

getDailyStats()

Returns an array of daily summaries, perfect for plotting historical trends in dashboards.

const stats = await telemetry.getDailyStats({ token, callerIp });

// Example Response
[
  { "date": "2023-10-23", "totalRequests": 980, "uniqueVisitors": 210 },
  { "date": "2023-10-24", "totalRequests": 1200, "uniqueVisitors": 280 },
  { "date": "2023-10-25", "totalRequests": 1500, "uniqueVisitors": 320 }
]

getVisitors()

Returns a map of all unique IPs and their detailed visit history for a specific date.

const visitors = await telemetry.getVisitors({ date, token, callerIp });

// Example Response
{
  "192.168.1.1": {
    "totalRequests": 2,
    "visits": [
      { "path": "/home", "timestamp": "2023-10-25T14:00:00.000Z", "responseTime": 45, "error": null },
      { "path": "/login", "timestamp": "2023-10-25T14:05:00.000Z", "responseTime": 120, "error": null }
    ]
  }
}

getVisitorByIp()

Retrieves the detailed telemetry data for a specific IP address on a given date.

const visitor = await telemetry.getVisitorByIp({ ip: "192.168.1.1", date, token, callerIp });

// Example Response
{
  "totalRequests": 1,
  "visits": [
    { "path": "/home", "timestamp": "2023-10-25T14:00:00.000Z", "responseTime": 45, "error": null }
  ]
}

countUniqueVisitors()

Returns the total number of unique IPs that visited the application on a given date.

const uniqueCount = await telemetry.countUniqueVisitors({ date, token, callerIp });

// Example Response
320

getPageViews()

Returns a record object mapping application paths to their total view count.

const views = await telemetry.getPageViews({ date, token, callerIp });

// Example Response
{
  "/home": 850,
  "/about": 320,
  "/pricing": 150
}

getTopPages()

Returns an array of the most visited pages, sorted by view count in descending order.

const top5 = await telemetry.getTopPages({ limit: 5, date, token, callerIp });

// Example Response
[ 
  { "path": "/home", "views": 1500 }, 
  { "path": "/about", "views": 420 },
  { "path": "/pricing", "views": 310 }
]

getPageByPath()

Retrieves detailed viewing statistics for a specific path, including the list of IPs that visited it and its average response time.

const pageData = await telemetry.getPageByPath({ path: "/pricing", date, token, callerIp });

// Example Response
{
  "path": "/pricing",
  "views": 310,
  "visitors": [
    "192.168.1.1",
    "10.0.0.5"
  ],
  "avgResponseTime": 65.3,
  "errorCount": 2
}

getAvgResponseTime()

Calculates the average server response time (in milliseconds) across all requests for a given date.

const avgMs = await telemetry.getAvgResponseTime({ date, token, callerIp });

// Example Response
42.5

getSlowRequests()

Returns a list of requests that exceeded a specified threshold in milliseconds, sorted from slowest to fastest.

// Find all requests that took longer than 500ms
const slow = await telemetry.getSlowRequests({ threshold: 500, date, token, callerIp });

// Example Response
[
  { "ip": "10.0.0.5", "path": "/checkout", "timestamp": "2023-10-25T14:03:00.000Z", "responseTime": 1250 },
  { "ip": "192.168.1.1", "path": "/api/report", "timestamp": "2023-10-25T15:10:00.000Z", "responseTime": 850 }
]

getErrors()

Returns a detailed list of all requests that resulted in an error (HTTP 500 status code).

const errorList = await telemetry.getErrors({ date, token, callerIp });

// Example Response
[
  { 
    "ip": "10.0.0.5", 
    "path": "/checkout", 
    "timestamp": "2023-10-25T14:03:00.000Z", 
    "responseTime": 1250, 
    "error": { "code": 500, "message": "Payment gateway timeout" } 
  }
]

countErrors()

Returns the total number of errors tracked by the telemetry on a given date.

const totalErrors = await telemetry.countErrors({ date, token, callerIp });

// Example Response
15

getTrafficSources()

Groups and counts traffic by source/medium (based on UTM parameters recorded via TrafficSource). You can filter by a specific source.

const sources = await telemetry.getTrafficSources({ source: "google", date, token, callerIp });

// Example Response
{ 
  "google/cpc": 450, 
  "google/organic": 200
}

flush()

Clears all telemetry data from the active store. Use with caution.

await telemetry.flush({ token, callerIp });

delete()

Deletes specific telemetry data based on IP address, date, or both.

// Delete all data for a specific date
await telemetry.delete({ date: "2023-10-25", token, callerIp });

// Delete all data for a specific IP across all dates
await telemetry.delete({ ip: "192.168.1.1", token, callerIp });

// Delete data for a specific IP on a specific date
await telemetry.delete({ ip: "192.168.1.1", date: "2023-10-25", token, callerIp });