Data Infrastructure
How Y2 ingests, normalizes, resolves, retains, and serves intelligence data
Y2 does not force every map layer into one universal record. It routes data according to its shape, update rate, and useful lifetime. This distinction matters when you compare source health, map freshness, search results, and API responses.
Architecture at a glance
The diagram shows three storage paths:
| Path | Best suited to | Examples |
|---|---|---|
| Normalized observations | Time-bound events that can be searched, grouped, and linked | Earthquakes, fires, disaster alerts, threat indicators, news events, Y2-extracted signals |
| Specialized caches | Data with a provider-specific shape or refresh cycle | Aircraft tracks, vessel positions, GPS interference, prediction markets, FININT indicators |
| Reference overlays | Curated context rendered directly by the app | Military bases, nuclear facilities, undersea cables, satellite anchors, CCTV hubs |
Reference overlays are a separate lane
A reference overlay can appear on the map without creating a normalized observation or a source-health record. Its presence does not imply that a provider fetch just succeeded.
From provider response to observation
Check the source circuit
Before a scheduled adapter calls its provider, it checks that source's circuit-breaker state. Five consecutive failures open the circuit for a five-minute cooldown. Other sources continue independently.
Normalize the provider payload
The adapter maps provider-specific fields to Y2 concepts such as source, title, category, severity, event time, coordinates, country, URL, fetch time, and expiry. The original provider payload may also be retained as source-specific metadata.
Resolve ontology references
Where the adapter has enough context, Y2 resolves or creates the related data source, place, entities, and incident. These links are optional: an observation can remain useful before all ontology references are available.
Upsert by source identity
Normalized observations use the pair (sourceType, sourceId) as their source identity. A later
fetch updates the active observation instead of blindly appending another row. Ambiguous legacy
ownership fails closed rather than overwriting an arbitrary record.
Recompute dependent intelligence
Writes linked to an incident enqueue corroboration work. Separate scheduled jobs classify observations, recompute conflict indicators, update statistical baselines, and reconcile stale incidents and markets.
Observation and ontology model
An observation is the evidence-bearing event record. It can carry:
- provider identity and raw metadata;
- event time, fetch time, and expiry;
- category, severity, coordinates, region, and country;
- provenance tier:
primary,derived, orsynthesized; - temporal meaning:
ongoing,past,forecast, oranalysis; - links to a canonical data source, place, entities, and incident.
The ontology supplies durable identity around those observations:
Entity aliases, provider identifiers, and canonical names have dedicated lookup indexes. Merge redirects preserve references when duplicate objects converge, while integrity jobs audit resolver indexes and candidate duplicates. Automated destructive entity merging is intentionally fail-closed until dependent references can be rewritten and verified.
Identity keys and safe deduplication
Y2 keeps durable object identity separate from the text used to find an object. A Convex document ID is the stable reference. Names, aliases, and provider identifiers are normalized into dedicated lookup projections:
| Identity path | Indexed lookup |
|---|---|
| Canonical name | entity kind + normalized canonical key |
| Alias | entity kind + normalized alias key |
| Provider identifier | entity kind + provider + normalized provider value |
| Ownership | object type + identity namespace + claim key |
Normalization applies Unicode compatibility normalization, trims and collapses whitespace, and compares case-insensitively while retaining the original display text. Candidate aliases are deduplicated before resolution.
These are typed compound indexes, not application-encoded byte keys. Convex owns physical ordering; Y2 chooses index fields in query order, uses indexed equality/range reads, and paginates large partitions. Reverse-time queries use numeric timestamps and descending index order instead of inverted timestamp strings. Opaque Convex IDs are never parsed for time or locality.
An index row is a rebuildable projection, not permission to choose an identity winner. Y2 can automatically remove a dead, stale, kind-mismatched, or same-owner duplicate projection. If a normalized key has multiple live owners, the resolver keeps both objects, skips the unsafe alias transfer, and records one review case. A merge occurs only through the governed reference-rewrite and verification process.
Agent Y2 signals
Completed Y2 reports can be processed into structured observations with extraction context, resolved entities, places, and incident links. Intel discovery and expansion jobs can create their own observations through the same ontology layer.
These synthesized records support investigation graphs, evidence links, search, and other intelligence views. They are not a promise that every future report automatically uses all prior observations as research context.
Freshness and retention
Freshness is controlled per data path rather than by one platform-wide timer.
| Data | Typical lifetime or lifecycle |
|---|---|
| Most normalized provider observations | Seven-day expiry, cleaned hourly |
| Aircraft tracks and GPS interference | 30 minutes |
| Naval vessel positions | 24 hours |
| Stock index cache | One hour |
| Prediction markets | Two-hour cache expiry, with separate market lifecycle reconciliation |
| FININT indicators | Provider-dependent, from two hours to 45 days |
| User Situation Room snapshots | 90 days |
| Inactive incidents | Automatically resolved after seven days without observations |
Y2 report, intel discovery, and intel expansion observations are excluded from the normal transient
observation cleanup because other evidence records can depend on them. Their own expiresAt value
does not make them subject to that hourly deletion path.
A cache is not an archive
API and map queries return the records that remain in their active storage path. If you need a durable external history, capture the relevant API results in your own governed store instead of relying on a short-lived Y2 cache.
How the app reads the data
The Situation Room combines several queries rather than loading one monolithic dataset:
- normalized observations are filtered by source, category, severity, geography, and event time;
- tracking and financial layers read their specialized tables;
- country views combine live calculations with curated country and infrastructure context;
- reference overlays are assembled from app registries and approved external display endpoints;
- the Sources page reads circuit-breaker records separately from map data.
This is why two surfaces can update at different times. For example, a source card can report a healthy provider while a specialized layer is empty because its rows expired, and a static overlay can remain visible while no runtime provider call exists for it.