Skip to main content

Datasets

A dataset is a named collection of reference data. It is what gives generated logs their content: without it, a template would render empty values.

Two kinds coexist, because they are edited the same way — an ordered collection of records — and differ only by the schema applied to each record.

Sample pool​

The data injected into templates. The engine draws a record from it for every generated event, and the keys present feed the editor's palette.

[
  { "process": "chrome.exe", "pid": 4412 },
  { "process": "svchost.exe", "pid": 890 }
]

A pool belongs either to a technology, or to nobody — it is then shared and any event type can point at it.

Records are not necessarily flat: the source corpus contains arrays and nested objects, which a template reaches through dotted notation (sample.answers[0].data).

Population​

A set of fictitious entities, gathering its four kinds in a single dataset — exactly as a config of the legacy project pointed at the same file name in conf/users/, conf/servers/ and conf/web/ — plus a fifth group, the roles, which says what each of them has installed.

{
  "users": [{ "name": "maurice.moss", "email": "maurice.moss@reynholm.com" }],
  "servers": [{ "name": "SRV-RDP-01", "ip": "192.168.2.10" }],
  "websites": [{ "host": "www.reynholm.com", "domain": "reynholm.com", "site_role": "marketing" }],
  "sites": [{ "city": "Paris", "country": "FR", "egress_ip": "193.51.110.149" }],
  "roles": [
    { "subject": "USER", "name": "*", "applications": ["microsoft-edge", "microsoft-365-apps"] },
    { "subject": "USER", "name": "IT Services", "applications": ["putty", "keepass"] },
    { "subject": "SERVER", "name": "database", "applications": ["microsoft-sql-server"] }
  ]
}

The keys are the ones templates expect: user.desktop_ip, server.agent_id, website.site_role. The editing table therefore shows literally what the template will see.

The geographic site, and the two public addresses​

A site is the office the employees of a city egress from: its egress_ip is the address they all share, as behind a real corporate NAT. It joins the user through their geo.city — no key to add to the users. A city pointing at no site is flagged as a warning, because that is the silent failure mode of a join on free text.

user.desktop_public_ip is therefore not “the address seen from the Internet”: site.egress_ip is. It is the address the workstation egresses from when it is not at the office — home or mobile network — the one a gesture marked off-site brings out.

user.desktop_ip, the third one, is the workstation's address on the internal network. It is not decorative: it is the one named by the log of a corporate machine the workstation joins — a 4624 for a remote session, a Kerberos ticket, access to a share. Those logs are written by the joined machine, which never sees the office NAT; announcing the public address there put a NetBIOS workstation name next to an Internet address, in a message addressed to a server on 10.x.

It is re-derived at import, and for a precise reason: the legacy project's values fell in the same /16 as the office egress address — that is the very field the egress ranges had been read from — so that a remote worker would have been indistinguishable from an employee at the office. Two populations even carried CloudFront, Cloudflare and Google addresses presented as an employee's. The derivation is deterministic: a db:seed does not replay the addresses. A country outside the catalogue keeps the imported value — we do not manufacture a French home for an Icelandic employee.

A population is already useful for the preview today: the template editor's “Render with” selector offers its users in place of the fixtures.

An incomplete entity is accepted, not refused​

The source corpus contains one degraded population — demo-msn-webinar provides only 9 of the 19 user keys. Refusing it would make its import impossible; accepting it silently would leave templates rendering empty. It is therefore accepted with a warning listing the missing keys, shown on the dataset's page.

Only the identity key is mandatory: name for a user or a server, host for a website.

A population lives somewhere​

A population carries a time zone (the “Metadata” card), and it is that zone which decides at what time its day is played: an action set to 08:00 in an archetype happens at 08:00 for those users. An IANA zone — Asia/Tokyo, America/New_York — and not an offset in hours: the offset is resolved for each simulated day, so daylight saving follows on its own.

Test benches relying on this population inherit it, and can override it case by case. With no setting, it is UTC. A sample pool, on the other hand, lives nowhere: the screen does not offer the field, and the API refuses it.

What each role has installed​

A role lists the applications installed on the devices of a slice of the population: a user's organisational unit (org.ou_name), a server's role (role). The name * is the slice's baseline — what every workstation, or every server, has — which each role adds to. A user from IT Services therefore has the baseline and the IT Services list; a server whose role nobody describes has the baseline only.

The list does two things, and it is the same list for both:

  • the software inventory declares it — the operating system, then exactly these applications, with the companions their installer brings (Chrome brings Google Update, Edge its updater, Acrobat its update service). Installed is not launched: a database server declares SQL Server even though none of its gestures starts it;
  • the logs launch nothing else. When a gesture names a process — a browser, an office application, a background task —, it can only be a Windows binary or one of these applications. An accounting workstation no longer launches AnyDesk if its role does not have it. No timestamp moves: the gestures stay the same, only the binary they name is chosen among what is installed.

An application is a key from the catalogue, which lives in the code: microsoft-edge, google-chrome, putty… A key the catalogue does not know is refused when saving — it would declare nothing and bound no process, so it can only be a typo. Two roles matching the same unit are refused as well (R&D and R and D are the same unit): only one of them would apply.

Three rules make the list safe to add:

  • a slice that declares no role keeps the previous behaviour: its inventory is what its gestures can launch, and its logs are unchanged to the bit. Declaring one server role does not empty the workstations;
  • Linux workstations are left out: the engine's process pools are Windows, the list does not apply to them, and their inventory comes down to their system;
  • an omission is shown, never silent: a role matching nobody, a unit or a server role that no entry describes, an application listed twice — each is a warning on the dataset's page, and the asset preview names the roles left without a list.

The six demonstration populations receive their roles when seeded, as modelling choices: the corpus holds no inventory to read them from. They are added only to a slice that has none, so a list edited on this screen survives the next seed.

The editor​

Two views of the same content — switching tabs loses nothing.

Table tab​

One row per record, one column per key. The columns are computed from the records present; “+ add a column” proposes a new one, which only materialises on the first entry.

A nested value (geo, org, answers) is edited as JSON inside its cell.

Emptying a cell removes the key

That is not the same as putting an empty string in it: templates tell the two apart. {{ sample.x | default('-') }} triggers on the absence of the key, never on "".

For a population, a filter shows one entity kind at a time — users, servers and websites do not have the same columns.

The Role filter does not show a table but one card per role: a Slice list, a Unit or server role field, and the catalogue's applications as checkboxes — a list typed as JSON in a cell would break at every keystroke. + Add a role adds a blank one: refused as long as it has no name, it stays on screen with the refusal above it, so that the name can be typed in.

A document refused but well formed — an entry just added, still missing its identity — keeps its table: only invalid JSON, or an unknown group, puts it on hold.

Beyond 50 records, the table paginates. That is rare: 247 of the corpus's 258 files hold fewer than 100, but nine carry more than 500.

JSON tab​

The whole document, in a code editor. That is the control the form actually submits — the table is an enhancement laid on top. Without JavaScript, editing therefore goes through the JSON, which stays fully functional.

Invalid JSON is flagged without overwriting the work in progress; the table simply waits. That is also where an application key outside the catalogue would be refused — the checkboxes offer only known ones.

Attachment to an event type​

The link is a foreign key, not a naming convention: two event types pointing at the same dataset share it explicitly. The corpus contains such a case — GitHub's pull-request-create and pull-request-merge have identical data.

Consequences:

  • deleting a dataset does not delete the event types that used it: they simply go back to having no data;
  • deleting an event type does not delete its dataset, which stays a named resource;
  • an event type recreated under the same name inherits nothing.

An event type without a dataset is a legitimate state: the rendering then receives {}, and the template's | default(…) calls do the rest.