Service: Logflare - EyevinnOSC/community GitHub Wiki

Logflare

Logflare (serviceId logflare-logflare) is a log ingestion and querying engine that stores and queries log events in a columnar database. In Eyevinn Open Source Cloud (OSC) it is a long-running service instance that uses a PostgreSQL database as its backend. License: Apache-2.0.

What it can do

  • Ingest log events from applications into named sources.
  • Query and browse log events in the Logflare web UI.
  • Receive logs from Node.js applications with the pino and pino-logflare libraries (example below).

Creating an instance

  1. Create a PostgreSQL instance (serviceId: "birme-osc-postgresql") with create-service-instance. It must be initialized with the SQL ALTER SYSTEM SET wal_level = 'logical';, set through the PostgresInitDbSql option when the instance is created, because it cannot be configured once the database is running. Wait until it is running and read its IP and port with describe-service-instance. The URI is postgres://postgres:<password>@<IP>:<PORT>/postgres.
  2. Store the URI with create-service-secret (serviceId logflare-logflare), for example name dburl.
  3. Create the instance with create-service-instance, serviceId: "logflare-logflare":
{ "name": "mylogs", "PostgresBackendUrl": "{{secrets.dburl}}", "SecretKeyBase": "{{secrets.logflaresecretbase}}" }

SecretKeyBase is required; the platform can generate a value for it. Wait for it to be running (wait-for-service-ready or get-instance-status) and get the URL with describe-service-instance.

Options

Option Required Type Meaning
name Yes string Name of the instance. Letters, digits and underscore only (^\w+$), at most 20 characters.
PostgresBackendUrl Yes string, sensitive PostgreSQL connection URI of the backend database, for example postgres://postgres:<password>@<IP>:<PORT>/postgres. The database user needs sufficient permissions to manage the schema. Corresponds to upstream POSTGRES_BACKEND_URL.
SecretKeyBase Yes string, sensitive Secret key base of the Logflare application. The platform can generate it.
DbSchema No string Database schema that scopes Logflare operations. The public schema is used when unset. Corresponds to upstream DB_SCHEMA.
DbEncryptionKey No string, sensitive Base64 encryption key used to encrypt sensitive data in the database. Corresponds to upstream LOGFLARE_DB_ENCRYPTION_KEY; upstream uses a default hardcoded key when none is set, so set your own.
ApiKey No string, sensitive Deprecated upstream alias (LOGFLARE_API_KEY) for the public access token.
PublicAccessToken No string If set, creates a public access token for the provisioned user, for ingestion or querying (LOGFLARE_PUBLIC_ACCESS_TOKEN, single-tenant mode).
PrivateAccessToken No string, sensitive If set, creates a private access token for the provisioned user, for management API use (LOGFLARE_PRIVATE_ACCESS_TOKEN, single-tenant mode).

Store sensitive options as service secrets and pass them as {{secrets.NAME}} (whole field value only). Overwriting a secret does not restart a running instance.

Using the instance

Open the instance URL and create a source with "New source" (for example myapp). Note the Source ID and the generated API key; both are shown once when the source is created. The API key is not recoverable afterwards, and anyone holding it can write log events into your source, so treat it as a secret.

Example Node.js application that sends logs:

mkdir myapp && cd myapp
npm init
npm install --save pino pino-logflare

myapp.js, where apiKey is the source API key, sourceToken the Source ID and apiBaseUrl your instance URL:

import pino from "pino"
import { createWriteStream } from "pino-logflare"

const stream = createWriteStream({
  apiKey: "<your-api-key>",
  apiBaseUrl: "https://<your-instance>.logflare-logflare.auto.prod-se.osaas.io",
  sourceToken: "<your-source-id>",
});

const logger = pino({}, stream)

logger.info("Informational message")
logger.error(new Error("things got bad"), "error message")

const child = logger.child({ property: "value" })
child.info("hello child!")

Run it with node myapp.js and the events appear in the source in the Logflare UI.

Limitations

  • The PostgreSQL wal_level must be set to logical at database creation time; it cannot be changed on a running instance.

Resources

⚠️ **GitHub.com Fallback** ⚠️