Generators and event types

A generator is a source technology: Palo Alto NGFW, Microsoft Entra ID, Office 365… It carries event types, and each event type carries a template describing the shape of a log from that technology.

The generator

Field	Role
Name	Functional identity, unique. The technical identifier is derived from it
Vendor	Informational only
Output format	Picks the validator applied to the preview
Alerts only	Excludes the technology from the continuous flow: it only emits on scenario

The source corpus is not only JSON. Five output shapes coexist:

Format	Example technology
JSON	Most of them — Office 365, GitHub, Google Workspace…
CEF	Fortinet FortiGate
CSV	Palo Alto NGFW, one line of 60+ positional fields
KEY_VALUE	Veeam Backup & Replication, between square brackets
TEXT	VMware ESXi and vCenter, Systancia Cleanroom — bare syslog

The event type

Field	Role
Name	In kebab-case. It used to be the name of the .jinja file in the source
Category	AUTH, PROCESS, NETWORK, FILE, DNS, ALERT or ASSET
Weight	Relative emission frequency. 0 = alerts only
Format	Inherited from the technology, unless explicitly overridden
Data	The dataset the engine draws a sample from

A zero weight is a state, not an absence

An event type with weight 0 is never emitted in the normal flow: it exists only to be triggered by an alert scenario. This is not a way of disabling it — it is a mode of operation in its own right, used by 178 of the corpus's 470 event types.

The format may depart from the technology's

Palo Alto NGFW emits CSV, but its dns event type is JSON. Four event types in the corpus depart this way. The event type then carries its own value; otherwise it inherits, and the interface shows “inherited”.

This is visible in the editor, on a badge: JSON — departs from CSV. The preview always validates against the effective format, never against the technology's.

The template editor

The template is Nunjucks, a direct port of the legacy project's Jinja2 — 467 of the 470 templates compile, the last three stumbling on a Python idiom.

The editor has three tabs:

• Event type — the source, in a syntax-highlighted editor, with a palette that inserts the snippet at the cursor position;
• Data — the attached dataset, and what its keys make available in the palette;
• Preview — the rendering, validated according to the format.

The palette

It offers the fields of the rendering context, grouped:

Group	Contents
User / Server	user.name, user.desktop_ip, server.agent_id…
Timestamp	timestamp.isoformat(), seven strftime formats, epoch in seconds and milliseconds
Identifiers	uuid4(), hexId(n), bigDigits(n), derivedId(…)
Randomness	randint, choice, weightedChoice, zpad, publicIp()
Structures	loops, conditionals, set, | tojson
Dataset fields	Computed from the keys actually present in the attached dataset

Random drawing is deterministic: with the same seed, two previews are identical. No Math.random enters the rendering.

The preview

It renders against a mock user, server and website — or, if a population exists, against a real entity picked in “Render with”. It then checks that the output honours the declared format:

• JSON must parse — the error is located by line and column;
• CEF must start with CEF: and carry at least seven header fields;
• CSV must fit on a single line and count at least 20 commas;
• KEY_VALUE must contain at least one key="value" pair;
• all formats: a {{ or a {% surviving the rendering is an error.

The warnings

They do not block saving, they flag:

• unguarded JSON value — "pid": {{ sample.pid }} renders "pid": when the key is missing, which breaks the JSON. A | default(0) is enough. This is the corpus's most frequent defect, hence the 934 | default(…) it contains;
• unknown context root — usr.name instead of user.name renders empty, silently;
• unknown filter.

A source that does not compile is refused outright: a broken template in the database would only be discovered at emission time, on a whole batch lost.

Sending the message

Below the preview, “Send this message” ships exactly the rendering shown to an intake. This is the only way to know whether a ported template is really accepted: a message that validates locally may very well be refused on arrival.

• Destination — one of the 7 SEKOIA regions, whose URL the product knows, or a custom URL (a test webhook, a local stub). The URL field stays visible at all times: it is only required for the custom destination.
• Intake key — passed to the intake in the request body, never stored in the database nor logged. It only travels through the queue, for the duration of the dispatch.

The posted body is the one of the SEKOIA API: {"intake_key": "…", "jsons": ["<the message>"]}.

An output the validator deems invalid can still be sent: the intake is precisely what we are trying to question, and its verdict is the only one that settles the matter.

The path is the real production path: the screen puts the message on the log-dispatch queue, api-sender consumes it and posts. What is displayed afterwards is the real result of the dispatch — HTTP status, duration, error message — not a queueing acknowledgement.

Status	What it says
Sent	The intake answered 2xx.
Failed	Definitive refusal (4xx), or 5 unsuccessful attempts against an outage (5xx, network).
Still in flight	Nobody consumed the queue in time. The message is not lost: it leaves as soon as api-sender is back.

A 5xx or a network cut is retried 5 times with exponential backoff; a 4xx is not — a rejected intake key will not fix itself by insisting.

“Latest dispatches of this event type” lists the last five dispatches of the event type currently open, not of the whole instance: on an editing screen, it is the history of what you are editing that informs. The global view lives on the Dispatches screen.

Deleting

Every event type and every generator carries a delete button, in two steps: the first click shows a confirmation that quantifies what goes (“and its 13 event types, templates included”), the second executes.

Deleting a generator does not destroy its datasets: they become shared and stay editable. Deleting an event type does not destroy its dataset either.