Importing the legacy project's corpus
A script populates the database from _source-oldproject, the Python application SekoiaCraft
replaces. It imports the technologies, their event types, their templates and their data.
pnpm db:seed [corpus-path]
The path is optional: failing that, _source-oldproject at the root of the repository.
This is the “reference data” step of getting started. Along the way the
command builds the packages the import depends on — it loads them by name, therefore from their
dist/. Without that build, the low-level equivalent
pnpm --filter @sekoiacraft/database seed:source fails on a module resolution until the repository
has been built.
_source-oldproject/ is in .gitignore — it contains the production intake keys of the SEKOIA
communities. Probing the corpus is therefore a local gesture, never a test or CI dependency. If
the folder is missing, the command does not stop for all that: it loads the
SQL snapshot versioned in the repository.
What it imports
| Source | Destination | Volume |
|---|---|---|
generators/__init__.py | Technologies (name, format, alerts only) | 28 |
<technology>.py | Event types, with their weights | 470 |
event/<source>/templates/*.jinja | Nunjucks templates | 470 |
event/<source>/data/*.json | Sample pools | 383 |
conf/{users,servers,web}/*.json | Populations | 6 |
The script requires python3: that is its only dependency. It really imports the registry's
classes rather than reading their source, because the event type lists are sometimes inherited and
the weights are computed, not declared.
Without the corpus, the snapshot
Those two requirements — the folder and Python — belong neither in a Docker image nor in a CI
runner. pnpm db:seed therefore chooses on its own: the corpus when it is present, and
otherwise the SQL snapshot from
packages/database/prisma/seed/corpus/, one file per technology, versioned in plain text. That is
how the deployed environments receive their technologies, their event types and their templates.
The snapshot is not a second source of truth: it is photographed from a database this very
import has just written. The corollary is worth remembering — refreshing the corpus, or correcting
the template rewriting rules, only takes effect in CI and in deployment after a
pnpm db:snapshot.
What it infers
None of this is written in the source: the script calls each generator and notes what it asks to render. The final report counts every inference, so that no approximation passes for a fact.
The template — the convention “an event type renders through the .jinja file bearing its name”
no longer holds. windows/process-created renders through security-4688.jinja, Microsoft IIS's 24
event types all go through iis-access.jinja, and Sekoia Agent picks its body according to the
entity's OS. The script observes the template actually reached; when an event type reaches several,
they are gathered behind an {% if %}.
The constants — the source generator computed outside the template what distinguishes two event
types sharing one file: remote-auth-success and local-auth-success both render through
security-logon.jinja and differ only by their envelope. Those values are laid down in a {% set %}
preamble, once per event type — nearly 5,000 lines across the whole corpus.
The category — the legacy project has no notion of it. It is inferred from the event type's name
(auth-failure → AUTH, dns-query-result → DNS, process-created → PROCESS…). Of the 470,
297 are inferred and 173 receive the default value NETWORK, to be corrected from the
interface.
The output format — it is read from the templates, on the body once the header tags have been
stripped. The majority rules at the technology level; the one that departs receives its own value —
today pan_ngfw/dns, the only JSON object of a generator emitting positional CSV.
The data attachment — an event type receives the pool the generator actually gives it,
identified by the file's content and not by its name: zscaler-event is fed by zscaler.json,
and seven Office 365 event types share message-trace.json. 420 of the 470 are attached this way;
the others go through the name, or have no data file in the source — which is normal.
What it sets aside
The source corpus is not the catalogue: some event types have no place in SekoiaCraft, whatever the
legacy project says. They are listed in
packages/database/prisma/seed/excluded-event-types.ts, and the final report names them — a
deliberate removal must be readable, not guessed from a count going down.
| Technology | Event type | Reason |
|---|---|---|
| Akamai WAF | waf-allow | Allowed traffic, dropped from the catalogue — 800 of the 850 weight points |
The exclusion cuts at both ends: no event type, and no data pool. It is carried here, and not by a
deletion from the interface, because the import is replayed on every seed:source — whatever is
only removed in the database would come back on the next pass. The script also purges what an
earlier import had created, so that an already-populated database converges.
It is replayable
Everything goes through upsert: a second run duplicates nothing. It does not overwrite a
template you may have edited by hand — unless explicitly asked:
pnpm db:seed --overwrite-templates
If a technology already exists under a name derived from its folder, it is renamed in place rather than duplicated: its datasets stay its own.
Starting from an empty database
The import completes a database; it does not clean it. To wipe and reload in one go:
pnpm db:reset [corpus-path] [--overwrite-templates]
Four chained steps: building the packages the seed depends on, prisma db push --force-reset
(dropping the database then the schema), purging the BullMQ queues, then this import — to which the
arguments are passed as-is.
The Redis purge erases the keys of the current prefix only (REDIS_QUEUE_PREFIX,
sekoiacraft-dev locally), never the whole of Redis: dev and preprod share the same instance. It is
necessary because the pending log-dispatch jobs reference dispatch batches that have just
disappeared.
The drop is irreversible and without confirmation. The script refuses to start if DATABASE_URL
or REDIS_URL do not point at localhost — a .env copied from .env.preprod therefore cannot
take the preprod database with it. Overriding this requires an explicit ALLOW_REMOTE_RESET=1.
What does not render yet
The import fills the database; it does not make everything functional for all that. Of the 470 event types:
| render and validate | 457 |
| render an invalid output | 13 |
| do not render | 0 |
| without a template | 0 |
The remaining thirteen all stumble on the same thing: Jinja2-specific idioms Nunjucks does not
know — filename.endswith(…), _map.get(…), rsplit, the dictionary literal {{{. They are fixed
one by one, in the template, from the editor.
The rendering context, however, is no longer missing anything: the 470 event types no longer reference a single root the engine does not expose. What the source generator computed on the outside — envelope constants, identifiers, values drawn from the entity or from the pool — is laid down as a preamble at import time.
No template refuses to compile today, but the safeguard remains: when a template does not compile,
it is stored as-is and reported (Broken templates). Losing it would deprive you of the very thing
to fix, and the editor locates the error by line.
The three that used to appear here — sekoia_agent/registry-value-set,
vmware_vcenter/lro-operation and vmware_vcenter/local-auth-failure — stumbled on the Python
slice (op_id[:8]), which Nunjucks cannot parse: its [ opens an indexed access, and the : is a
syntax error there. It is now translated to truncate at import. The same pass fixed Python's %
formatting, more insidious since it compiled: in JavaScript % is a modulo, so that '%03d' % 45
rendered NaN in the timestamp of 32 vCenter and Salesforce templates, without any validation
noticing — the output stayed valid JSON.