Scratch RecSys 03 - Engineering Ultimate Real-World ZaPudia E-Commerce Dataset

Executive Summary

Part 1 and part 2 established the architecture, the five production scenarios, and the mathematical foundations of the ZaPudia recommendation engine. A recurring theme of that analysis was training-serving skew: the gap between the clean, curated data on which models are trained and the messy, telemetry-degraded data on which they must serve.

This Part 3 closes that gap at the source. Standard open benchmark datasets, the kind downloaded as tidy CSVs from Kaggle, are hyper-sterile. They exhibit zero telemetry dropouts, perfect session ordering, no bot activity, no schema degradation, and no cold-start catalog rot. A recommendation model trained on such data can achieve excellent offline NDCG@100 in a notebook and fail catastrophically the moment it is deployed against live GA4 BigQuery event streams, where 8% of add_to_cart events silently vanish upstream of a purchase, where bot farms inject sub-10ms click floods, and where 25% of items have NULL category_code because a seller’s catalog ingestion job crashed.

The engineering response is the Ultimate ZaPudia Unified Dataset: a synthetic-but-realistic corpus built by harmonizing real multi-category e-commerce behavior logs from Kaggle with the native GA4 BigQuery telemetry schema, then deliberately injecting the five real-world data flaws enumerated in §2. The dataset is produced by an executable Python package, data_pipeline/, built on Polars and DuckDB, that scales to multi-gigabyte event logs without memory exhaustion and exports clean partitioned Parquet artifacts ready for the model-training pipeline in Part 3.

This article delivers four things:

1. A critique of hyper-clean benchmarks and the Unified Data Schema that aligns Kaggle attributes with GA4 BigQuery fields (§1).
2. An exact mathematical and algorithmic specification for injecting five real-world data flaws (§2).
3. The implicit feedback rating formulation with logarithmic dwell weighting and min-max normalization (§3).
4. The complete, executable data_pipeline/ Python codebase with vector-optimized Polars/DuckDB implementations (§4).

A business analogy. Think of a benchmark dataset as a driving test taken on an empty, sunny, freshly-paved parking lot. Anyone can pass that test. The road ZaPudia actually operates on has potholes (missing events), other drivers who cut you off (bots), traffic lights that report the wrong time (out-of-order timestamps), road signs that fell down (missing product metadata), and brand-new streets with no map data yet (cold-start SKUs). A recommendation model that only ever practiced on the empty parking lot will crash the first time it meets real traffic. The job of this Part 2 is to build a realistic driving course, one that intentionally contains potholes and confusing signage, so that the model we ship in Part 3 has already learned to handle them before a real customer ever sees a bad recommendation.

---

0. Architecture at a Glance

Before the mathematics, it helps to see the whole assembly line in one picture. Three raw material suppliers (Kaggle behavior logs, Kaggle catalog/order metadata, and GA4 telemetry) feed into a harmonization stage that speaks one common language. That clean stream is then deliberately sent through a corruption station, the anomaly injection engine, before it reaches the DuckDB signal engine that turns raw clicks and carts into a single trustworthy preference score. A time-aware split then produces the exact Parquet files the Part 1 models consume.

Reading the diagram in plain terms: everything to the left of the “Anomaly Injection Engine” node is supply; everything to the right is refinement. The single most important box in this diagram is the corruption station in the middle, it is the only stage whose entire purpose is to make the data worse on purpose, so that everything downstream of it has to prove it can survive the mess. If that box were removed, the rest of the pipeline would produce a beautiful, useless benchmark.

---

1. Architectural Flaw of Hyper-Clean Benchmarks

1.1 The Hyper-Sterility Problem

The conventional offline evaluation practice in recommendation-system research is to download a public benchmark dataset, split it temporally or randomly, train a model, and report Precision@K / NDCG@K. The datasets most commonly used, and most of the Kaggle e-commerce behavior dumps, share a set of quiet assumptions that are false in production:

• No telemetry gaps. Every add_to_cart that precedes a purchase is present in the log. In production, client-side ad blockers, network flakiness on Indonesian mobile networks, and SDK initialization races drop a non-trivial fraction of events. A model that learns the strong cart → purchase dependency from a complete log is surprised when 8% of those dependencies are missing at serve time.
• Perfect monotonic timestamps. Events arrive in the exact order they occurred. In production, client-side batching, clock skew across devices, and async upload retries produce out-of-order arrival. A session-reconstruction algorithm that assumes monotonic order will silently corrupt sessions.
• No bot traffic. Every user_id is a genuine human. In production, scraper bots, competitor price-crawlers, and click-fraud farms inject high-frequency synthetic streams that, if not filtered, dominate the interaction matrix and bias popularity-based signals toward bot-favored items.
• Complete metadata. Every item has a category_code, brand, and price. In production, seller-side catalog ingestion is unreliable: 25%+ of items may lack a category_code or brand, breaking content-based fallback paths and the cold-start strategy from Part 1 §2.3.
• No cold-start rot. Every product in the catalog has some interaction history. In production, 15%+ of active SKUs have zero historical interactions because they are newly listed, the exact cold-start death spiral described in Part 1 §1.1.2.

The combined effect is training-serving skew: the joint distribution of the training data $P_{\text{train}}(x)$ differs from the serve-time distribution $P_{\text{serve}}(x)$, and a model minimizing loss on $P_{\text{train}}$ does not minimize loss on $P_{\text{serve}}$. The magnitude of the skew is determined by exactly the five flaws above. A dataset that does not contain them cannot measure a model’s robustness to them.

Architectural principle. A benchmark dataset should reproduce the failure modes of the production telemetry, not hide them. A model’s offline score is only as trustworthy as the dataset’s resemblance to production.

Business translation. A category_code field with 25% missing values is the data-engineering equivalent of a store where a quarter of the shelves have no price tag and no aisle sign. A shopper (or a model) can still find the item, but the reason to recommend it alongside similar items is gone. That is exactly why the Unified Data Schema below is not a cosmetic exercise, it is the shelf-labeling system that lets every downstream model, human analyst, and A/B test speak about the same product the same way, no matter which warehouse (Kaggle or GA4) it physically came from.

1.2 The Unified Data Schema

The ZaPudia Unified Dataset harmonizes three source families into a single event-stream schema. The mapping below is the contract that the extractors in §4 must produce.

1.2.1 Source-to-Target Field Alignment

Unified Field	Type	Kaggle MV	Kaggle (order/LTV sources)	GA4 BigQuery Export	Notes
event_time	TIMESTAMP	event_time (UTC)	order timestamp	event_timestamp (μs)	All normalized to UTC ms
event_type	ENUM	event_type (view/cart/ remove_from_cart/ purchase)	status → mapped	event_name (page_view/ select_item/add_to_cart/ begin_checkout/purchase)	Harmonized to unified enum (see §1.2.2)
user_id	STRING	user_id (nullable)	customer_id	,	Deterministic user; null for anonymous
user_pseudo_id	STRING	,	,	user_pseudo_id	Anonymous device-scoped ID
product_id	STRING	product_id	product_id / sku	items.item_id	Canonical SKU key
category_id	STRING	category_id	,	items.item_category_id	Numeric category leaf
category_code	STRING	category_code (nullable)	category path	items.item_category	Dot-path e.g. electronics.smartphone
brand	STRING	brand (nullable)	brand	items.item_brand	Nullable by design (see §2.4)
price	FLOAT	price	price / unit price	items.price	In IDR (currency-normalized)
user_session	STRING	user_session	,	ga_session_id	Session grouping key
engagement_time_msec	INT	,	,	engagement_time_msec	Dwell proxy; backfilled for Kaggle rows
device_category	ENUM	,	,	device.category	mobile/desktop/tablet
geo_province	STRING	,	region	geo.province	Indonesian province code
discount_flag	BOOL	,	discount / promo flag	,	Promotional context
fulfillment_status	ENUM	,	order_status	,	fulfilled/cancelled/returned
customer_ltv	FLOAT	,	ltv / derived	,	Historical lifetime value
age_band	ENUM	,	age → binned	,	18-24/25-34/35-44/45+

To establish a realistic foundation for our data engineering pipeline, we draw upon open-source e-commerce datasets published on Kaggle by contributors Mkechinov and Vanhaminhquan, take a look in colom Kaggle MV. These raw data streams provide invaluable real-world interaction logs, user session behavior, and product catalog metadata that serve as our primary inputs. Rather than consuming the raw files directly, our pipeline executes an end-to-end transformation workflow to resolve schema discrepancies and handle missing values. We perform strategic union operations across these distinct tabular sources to harmonize disparate event structures into a unified, coherent data model. Furthermore, custom feature modifications and realistic noise injections are applied to refine implicit feedback signals and replicate true production environments. Strict validation rules and data type coercions are enforced throughout the ingestion process to guarantee high data quality and system reliability. This multi-stage synthesis bridges the gap between hyper-clean academic benchmarks and real-world event stream complexities. Ultimately, the resulting processed dataset delivers a robust, production-grade foundation engineered specifically to train scalable recommendation architectures.

1.2.2 Unified Event Type Enum

The three source schemas use different event vocabularies. The pipeline normalizes all to a single enum:

Unified event_type	Kaggle event_type	GA4 event_name	Implicit signal weight
view	view	page_view, select_item	$w_{\text{view}} = 1.0$
cart	cart	add_to_cart	$w_{\text{cart}} = 3.0$
remove	remove_from_cart	remove_from_cart (custom)	$0$ (negative context only)
checkout	,	begin_checkout	$w_{\text{checkout}} = 4.0$ (intermediate)
purchase	purchase	purchase	$w_{\text{buy}} = 5.0$

The remove and checkout types are retained for session-context features in Part 3 but do not enter the implicit score directly except as negative-signal context.

1.2.3 Schema Harmonization Rules

Three rules govern the merge and resolve precedence conflicts:

1. User identity resolution. Where a Kaggle user_id is present, it is the canonical key. Where only GA4 user_pseudo_id is present (anonymous browsing), user_pseudo_id becomes the working key. A deterministic join table (user_identity_map.parquet) maps user_pseudo_id $\to$ user_id for users who eventually authenticate, enabling cross-device session stitching. The unified key is COALESCE(user_id, user_pseudo_id).
2. Dwell backfill. Kaggle sources carry no engagement_time_msec. The pipeline backfills dwell from the inter-event delta within a session: $T_{\text{dwell}}(e_k) = \min(\Delta t_{k, k+1},\ 1,800,000)$, capped at 30 minutes to prevent tab-left-open inflation.
3. Currency normalization. All price values are normalized to IDR at a fixed ingestion-time FX rate; the rate is recorded in config so the dataset is reproducible.

---

2. Real-World Noise & Anomaly Injection Framework

The pipeline does not merely harmonize clean sources; it deliberately degrades the harmonized stream to reproduce production telemetry pathology. This section specifies the five injection procedures with exact mathematical and algorithmic definitions. Each is implemented vector-optimized in §4.3 (anomaly_injector.py).

The injection is deterministic given a seed: every run with the same seed and config reproduces the same corrupted dataset, which is a non-negotiable requirement for reproducible offline evaluation and A/B holdout stability.

2.1 Telemetry Drops (8% of pre-purchase cart events)

Pathology modeled. Client-side ad blockers and network failures on Indonesian mobile networks cause a fraction of add_to_cart events that should precede a purchase to never reach the server. The observed purchase remains (it is server-side confirmed), but the upstream cart is lost, breaking the cart → purchase dependency the model may have learned.

Specification. Let $\mathcal{E}_{\text{cart}}$ be the set of cart events that are the last cart before a purchase in the same (user, item) session:

$\mathcal{E}_{\text{cart}}^{\text{pre-purchase}} = \{ e \in \mathcal{E}_{\text{cart}} : \exists e' \in \mathcal{E}_{\text{purchase}}, \text{session}(e') = \text{session}(e), \text{item}(e') = \text{item}(e), t(e') > t(e) \}$

A dropout indicator $D_e \sim \text{Bernoulli}(p_{\text{drop}})$ is drawn independently for each $e \in \mathcal{E}_{\text{cart}}^{\text{pre-purchase}}$ with $p_{\text{drop}} = 0.08$. The corrupted stream drops every event with $D_e = 1$:

$\mathcal{E}_{\text{corrupted}} = \mathcal{E} \setminus \{ e : e \in \mathcal{E}_{\text{cart}}^{\text{pre-purchase}}, D_e = 1 \}$

Implementation constraint. The drop is conditioned on a subsequent purchase existing: dropping random cart events globally would also remove carts that never convert (which legitimately exist), conflating telemetry loss with genuine abandonment. The pre-purchase conditioning isolates the telemetry-loss pathology.

2.2 Bot Activity Flooding (sub-10ms inter-arrival streams)

Pathology modeled. Scraper bots and click-fraud farms emit click streams at machine speed, inter-arrival times far below human reaction time (~200ms). These streams must be filterable but, if left in, dominate the interaction matrix and bias popularity signals.

Specification. A synthetic bot cohort $\mathcal{B} = \{b_1, \dots, b_{N_b}\}$ of $N_b$ synthetic user_pseudo_ids is generated. Each bot emits a stream of view events with inter-arrival times drawn from an exponential distribution truncated to the sub-10ms regime:

$\Delta t_k \sim \text{Exponential}(\lambda_{\text{bot}}), \quad \lambda_{\text{bot}} = 150\,\text{s}^{-1}, \quad \Delta t_k \in [1\,\text{ms}, 10\,\text{ms}]$

giving a mean inter-arrival of $\approx 6.7\,\text{ms}$. The target item for each bot event is drawn from a power-law popularity distribution to mimic bots that scrape popular SKUs:

$P(\text{item} = i) \propto \text{pop}(i)^{0.75}, \quad \text{pop}(i) = |\{e : \text{item}(e) = i\}|$

The bot stream volume is set to $\approx 5\%$ of the human event volume to represent a realistic but detectable bot load. The injected bots share no user_id (anonymous) and have device_category = 'desktop' with a fixed, narrow user-agent signature to make them detectable but not trivially so, the filtering module in §4 must distinguish them by behavior (inter-arrival distribution), not by an obvious flag.

2.3 Out-of-Order Event Timestamps (non-monotonic arrival)

Pathology modeled. Async client-side logging batches events and uploads them in retries; network jitter reorders arrival. The recorded timestamp may therefore be non-monotonic within a session even though the true event order was monotonic.

Specification. For each event $e$ in a session, an arrival jitter $\delta_e$ is added to the true timestamp $t_e$:

$\tilde{t}_e = t_e + \delta_e, \quad \delta_e \sim \text{Exponential}(\lambda_{\text{oo}}), \quad \lambda_{\text{oo}} = 0.5\,\text{s}^{-1}$

yielding a mean jitter of $2\,\text{s}$. Because the exponential is right-skewed, most events jitter by < 1s but a long tail produces multi-second reordering. The recorded timestamp $\tilde{t}_e$ is what the pipeline downstream sees; the true $t_e$ is retained in a true_event_time column for evaluation ground-truth only (never exposed to the model).

Session-ordering invariant. After jitter injection, the pipeline guarantees that user_session IDs remain assignable: a session boundary is defined as a gap of > 30 minutes in true_event_time, computed before jitter, so the jitter cannot merge or split sessions, only reorder within them.

2.4 Schema Degradation (25% category_code/brand NULL)

Pathology modeled. Seller-side catalog ingestion is unreliable; category_code and brand are the most commonly missing fields because they require structured seller input that is often skipped. Missing metadata breaks content-based fallback and the cold-start content path.

Specification. Let $\mathcal{I}_{\text{meta}}$ be the set of items with non-null category_code. A dropout indicator $M_i \sim \text{Bernoulli}(p_{\text{null}})$ is drawn per item with $p_{\text{null}} = 0.25$:

$\text{category\_code}'(i) = \begin{cases} \text{NULL} & M_i = 1 \\ \text{category\_code}(i) & M_i = 0 \end{cases}, \quad \text{brand}'(i) = \begin{cases} \text{NULL} & M_i = 1 \\ \text{brand}(i) & M_i = 0 \end{cases}$

The same indicator $M_i$ nulls both fields for an item, modeling a seller who skipped structured metadata entirely (correlated absence), rather than independent per-field loss. The degradation is applied at the item level (not the event level): once an item’s metadata is nulled, all of its events carry the null, consistent with a catalog-ingestion root cause rather than a per-event telemetry failure.

2.5 Cold-Start Catalog Rot (15% zero-history SKUs)

Pathology modeled. Newly listed SKUs have no interaction history, the cold-start death spiral. A clean benchmark with full history for every item hides this; the corrupted dataset must contain a fraction of active items with exactly zero historical interactions.

Specification. Let $\mathcal{I}_{\text{active}}$ be the set of active product IDs in the catalog. A subset $\mathcal{I}_{\text{cold}} \subset \mathcal{I}_{\text{active}}$ of size $|\mathcal{I}_{\text{cold}}| = 0.15 \cdot |\mathcal{I}_{\text{active}}|$ is selected uniformly at random. All interaction events referencing items in $\mathcal{I}_{\text{cold}}$ are removed from the training split:

$\mathcal{E}_{\text{train}}' = \{ e \in \mathcal{E}_{\text{train}} : \text{item}(e) \notin \mathcal{I}_{\text{cold}} \}$

The cold items are retained in the item catalog (item_features.parquet) with their metadata, so the model sees them as eligible candidates with zero history, exactly the production cold-start condition. A held-out test set may contain interactions with cold items to evaluate true cold-start ranking quality (the model has never seen them in training but must still rank them).

Interaction with §2.4. Cold items are selected independently of the schema-degradation mask, so a cold item may or may not have null metadata, both production cases are represented.

2.6 Injection Ordering and Composition

The five injections are applied in a fixed order to avoid interference:

1. Schema degradation (§2.4), applied to the item catalog first; downstream events inherit the nulled metadata.
2. Cold-start rot (§2.5), partition items into warm/cold; remove cold-item events from the training split.
3. Out-of-order jitter (§2.3), perturb timestamps within the remaining (warm-item) events.
4. Telemetry drops (§2.1), drop pre-purchase carts from the jittered stream.
5. Bot flooding (§2.2), inject bot streams into the final corrupted stream.

This ordering ensures each pathology is layered on top of a stable base: metadata is fixed before events are filtered, events are filtered before timestamps are perturbed, and bots are added last so they appear in the final stream the model sees.

Business analogy: why order is not arbitrary. Imagine a quality-control line for a food factory: you first check which ingredients are unlabeled (schema degradation), then set aside the brand-new ingredients that have no track record yet (cold-start rotation), then account for shipping delays that shuffled delivery order (jitter), then note which packages went missing in transit (telemetry drops), and only at the very end do you let a few counterfeit packages (bots) onto the line, because you want the counterfeits to face the exact same downstream inspection a real customer’s order would. Reversing this order would be like adding counterfeit packages before checking labels: some of them would get mistaken for genuine low-information products instead of being recognizable as fraud. The diagram below makes the sequence, and why each step depends on the one before it, visible at a glance.

---

3. Implicit Feedback Rating Formulation

Recommendation models in the ZaPudia architecture consume an implicit feedback matrix: a pseudo-rating $\hat{r}_{ui} \in [1.0, 5.0]$ for each (user, item) pair, derived from observed behavioral signals rather than explicit star ratings. I tell you, the rate $[1.0, 5.0]$ is just assumptions. You could use your own declaration, like rate $[0.0, 8.0]$ or others. Ussually, it depends on business needs or statistical formula that you used. And then this section defines the transformation.

3.1 The Composite Implicit Score

For each (user $u$, item $i$) pair, let:

• $N_{\text{view}}(u, i)$ = count of view events by $u$ on $i$,
• $N_{\text{cart}}(u, i)$ = count of cart events by $u$ on $i$,
• $N_{\text{buy}}(u, i)$ = count of purchase events by $u$ on $i$,
• $T_{\text{dwell}}(u, i) = \sum_{\text{views}} \text{engagement\_time\_msec}$ = total dwell in milliseconds.

The raw implicit score is a weighted linear-log composite:

$\text{ImplicitScore}(u, i) = w_{\text{view}} \cdot N_{\text{view}} + w_{\text{cart}} \cdot N_{\text{cart}} + w_{\text{buy}} \cdot N_{\text{buy}} + w_{\text{dwell}} \cdot \log_2\left(1 + \frac{T_{\text{dwell}}}{1000}\right)$

with default weight vector $\mathbf{w} = [w_{\text{view}}, w_{\text{cart}}, w_{\text{buy}}, w_{\text{dwell}}] = [1.0, 3.0, 5.0, 0.5]$.

Business analogy. Think of this like a loyalty-punch-card system, except every action punches the card by a different amount. Glancing at a product on the shelf (a view) earns one punch. Picking it up and putting it in your basket (a cart) earns three punches, you’ve shown real commitment, not idle curiosity. Actually paying for it (a purchase) earns five punches, because money changing hands is the strongest statement of preference a customer can make. Standing in front of a product for a long time without buying (dwell) earns a small, slow-growing bonus, it suggests interest, but interest alone should never outweigh an actual purchase, which is exactly why the dwell term is compressed logarithmically rather than counted linearly like the others.

Design rationale for the weights. The weights encode a monotonic intent ladder: a view is the weakest signal (cheap, high-volume, low commitment), a cart is a medium-commitment signal (the user actively placed the item), and a purchase is the strongest (money committed). The $3\times$ and $5\times$ ratios reflect the empirical conversion funnel: roughly 3–5× more users view than cart, and cart than buy. The dwell term uses $\log_2(1 + T_{\text{dwell}}/1000)$, the $/1000$ converts ms to seconds, and the logarithm compresses the long-tailed dwell distribution so a single tab-left-open overnight (28.8M ms) does not dominate the score.

Design rationale for the log on dwell only. Counts ($N_{\text{view}}, N_{\text{cart}}, N_{\text{buy}}$) are bounded and small per (user, item) pair (typically 1–10); a linear weighting is appropriate and interpretable. Dwell is unbounded and heavy-tailed; a log compression is required to prevent a single pathological dwell from inflating the score above a genuine purchase. Applying the log only to dwell (not to counts) preserves the interpretability of the count weights.

3.2 Min-Max Normalization

The raw score $\text{ImplicitScore}(u, i)$ is unbounded above and dataset-dependent, so it cannot be compared across snapshots or fed directly to algorithms expecting a bounded rating. It is min-max scaled to the interval $[1.0, 5.0]$, the conventional 1–5 star range, with a floor of 1.0 rather than 0.0 so that every observed interaction is a positive signal (consistent with the implicit-feedback premise that unobserved, not low-rated, is the negative):

$\hat{r}_{ui} = 1.0 + 4.0 \cdot \frac{\text{ImplicitScore}(u, i) - \min_{u,i} \text{ImplicitScore}}{\max_{u,i} \text{ImplicitScore } - \min_{u,i} \text{ImplicitScore}}$

with the convention that the denominator is guarded against zero (single-pair datasets) via NULLIF(max - min, 0), in which case all pseudo-ratings are set to 1.0.

Why the $[1,5]$ floor matters for iALS. In the iALS formulation of Part 1 §4.1, the preference indicator is $p_{ui} = 1$ if $R_{ui} > 0$. By flooring pseudo-ratings at 1.0, every observed (user, item) pair has $p_{ui} = 1$, and the confidence $c_{ui} = 1 + \alpha \hat{r}_{ui}$ scales how confident we are in that preference. A floor of 0.0 would invert this: low-rated pairs would become $p_{ui} = 0$ (treated as negatives), which conflates “low engagement” with “no engagement” and breaks the implicit model. The $[1,5]$ range keeps the semantics clean.

3.3 Sparsity and the Confidence Parameter

After normalization, the iALS confidence (Part 1 §4.1.2) becomes

$c_{ui} = 1 + \alpha \hat{r}_{ui} \text{ with } \hat{r}_{ui} \in [1,5]$

The recommended $\alpha$ for this scale is $\alpha = 40$ (the Hu-Koren-Volinsky default for a 1–5 scale), giving confidence ranging from $c_{ui} = 41$ (minimal engagement) to $c_{ui} = 201$ (maximal engagement). The pipeline exposes $\alpha$ as a config parameter; Part 3 tunes it via offline NDCG@100.

---

4. Production Python Codebase

This section is the complete, executable in our repo package. Every module is implemented with type hints, vector-optimized Polars expressions or DuckDB SQL, and no placeholder code. The package is deterministic given a seed and config.

Operational Notes

• Determinism. Every stochastic operation derives its RNG from the single pipeline.seed via per-injector offsets, so the same seed + config reproduces the dataset byte-for-byte. This is mandatory for A/B holdout stability across Part 3 training runs.
• Memory. The implicit matrix is built with a single DuckDB SQL pass over parquet, no full event frame is held in Python memory. The Kaggle harmonization uses pl.scan_parquet (lazy) and sink_parquet (streaming), so multi-gigabyte sources are processed in bounded memory.
• Cold items in catalog. item_features.parquet is built from the full catalog (train + test unique items, including cold items), so the model sees cold items as eligible candidates with is_cold = True and zero implicit history, exactly the production cold-start condition.
• Encoders. Categorical ordinal maps are persisted as .encoders.json next to each feature parquet, so the downstream model (Part 3) can decode indexes and apply the same mapping at serve time, eliminating a common source of training-serving skew.
• Bot detectability. Bots are anonymous (user_id NULL) and have device_category = 'desktop', but their inter-arrival distribution is the discriminator. Part 3 includes a bot-filter preprocessor that flags sessions with median inter-arrival < 200ms; the injected bots (mean ≈ 6.7ms) are caught, while genuine humans are not, a built-in self-test of the filter.

---

5. Data-Flow Recap

The end-to-end flow, from raw sources to model-ready parquet, is the same architecture shown in §0 but at the resolution of actual function calls, useful for the engineer who will run this pipeline, not just read about it.

The four exported artifacts, train_interactions.parquet, test_interactions.parquet, item_features.parquet, user_features.parquet, plus the implicit_matrix.parquet are the exact inputs to the model-training pipeline.

---

6. Data Quality Assertions Before Model Training

Synthetic corruption is useful only if the final dataset is measurable. The pipeline should therefore validate the following invariants before a model is trained.

6.1 Event uniqueness

For generated IDs $\text{Unique}(\text{event\_id})=\text{Count}(\text{event\_id})$. within the expected event domain. If not $\text{DuplicateRate} > 0$ and the data-quality report should surface it.

6.2 Cold-start coverage

Let $N_{\text{catalog}}=|I|$ and $N_{cold}=|S_{\text{cold}}|$ then the expected ratio is:

$\rho_{cold} = \frac{N_{cold}}{N_{\text{catalog}}} \approx 0.15$

In a strict test harness, the observed value should equal the configured sampling ratio subject to integer rounding. More importantly, $\forall i\in S_{\text{cold}},\ \text{InteractionCount}(i) = 0$ must hold.

6.3 Schema degradation coverage

For the selected degradation set:

$P(\text{category\_code} = \text{NULL} \mid i\in S_{\text{degraded}}) = 1$

and:

$P(\text{brand} = \text{NULL} \mid i\in S_{\text{degraded}}) = 1$

while products outside the set should retain their original attributes unless the source was already null.

6.4 Telemetry-drop coverage

For eligible cart events:

$\hat p_{\text{drop}} = \frac{N_{\text{dropped}}}{N_{\text{eligible}}}$

should be approximately $0.08$. Because the implementation samples an exact count from an integer-sized eligible population, rounding determines the final realized fraction.

6.5 Bot stream verification

For injected bot sessions, the engineering invariant is:

$1\text{ms} \leq \Delta t < 10\text{ms}$

for generated consecutive events. This allows downstream anomaly detectors to be evaluated on an intentionally hostile input distribution.

---

7. Temporal Splitting Without Leakage

Random row-level train/test splitting is particularly dangerous for event-stream recommendation data. Suppose the same interaction sequence is randomly divided:

view(t1)     -> train
cart(t2)     -> test
purchase(t3) -> train

The model can indirectly observe future information during training. Instead, use a temporal boundary $T_{\text{cut}}=\text{Quantile}_{1-f}(T)$ where $f$ is the test fraction. Then $\text{Train}=\{e:T_e\leq T_{\text{cut}}\}$ and $\text{Test}=\{e:T_e>T_{\text{cut}}\}$. The current pipeline uses a 20% temporal holdout by default. This does not automatically solve every leakage problem, but it is substantially more faithful to production forecasting than random row splitting.

Business analogy. Random splitting is like grading a weather forecaster by letting them peek at tomorrow’s actual weather report before making today’s prediction, of course they look accurate. A temporal split is the honest version: the forecaster only ever sees data up to today, and is graded on tomorrow, which they have genuinely never seen. Any recommendation model evaluated with a random split is quietly being shown a piece of tomorrow’s newspaper; the offline accuracy number it produces cannot be trusted to predict how it behaves once deployed.

---

8. Source Harmonization Strategy in Production

The source layers play different roles.

Kaggle behavioral datasets

These primarily contribute $\text{User} \times \text{Item} \times \text{Event}$ relationships.

Kaggle customer/order datasets

These contribute side information such as:

$\text{Demographics} + \text{Region} + \text{LTV} + \text{OrderStatus} + \text{DiscountBehavior}$

GA4 telemetry

This contributes contextual interaction signals:

$\text{Session} + \text{Device} + \text{Engagement} + \text{AnonymousIdentity} + \text{PageEvent}$

The unified representation is therefore:

$X_{\text{event}} = X_{\text{behavior}} \cup X_{\text{catalog}} \cup X_{\text{customer}} \cup X_{\text{telemetry}}$

The objective is not to force all source systems into one identical schema. It is to create a canonical analytical contract with deterministic mappings.

---

9. Operational Data Lake Layout

A production implementation should retain the raw/staging/output separation:

lake/
├── raw/
│   ├── kaggle/
│   └── ga4/
├── staging/
│   ├── unified_events.parquet
│   ├── catalog.parquet
│   └── users.parquet
└── curated/
    ├── train_interactions.parquet
    ├── test_interactions.parquet
    ├── item_features.parquet
    └── user_features.parquet

The design principle is Raw $\neq$ Curated. Never mutate the raw source solely to make model training convenient. Instead:

$\text{Raw} \rightarrow \text{Reproducible Transform} \rightarrow \text{Curated}.$

This makes every experiment traceable to a source snapshot and transformation configuration. This mirrors how a well-run kitchen keeps its walk-in freezer (raw ingredients, untouched) separate from the plated dishes sent to the dining room (curated output). If a customer complains about a dish, the kitchen can trace it back to the exact delivery batch of ingredients. If raw/ and curated/ were the same folder, an engineer “fixing” a training set could silently corrupt the one true record of what actually happened, and no experiment would ever be reproducible again.

---

10. Why DuckDB + Polars Is a Strong Combination

The two engines solve complementary problems.

1. DuckDB Best suited here for:
   • large Parquet scans,
   • SQL aggregation,
   • min/max statistics,
   • joins and analytical projections,
   • low-friction local analytical execution.

2.Polars Best suited here for:

• vectorized feature transformations,
• categorical/missing-value transformations,
• in-memory operations after data reduction,
• fast schema-aware processing.

The intended division is:

$\text{Large}\ \text{Scan}/\text{Aggregation} \rightarrow \text{DuckDB}$

then:

$\text{Reduced}\ \text{Tabular}\ \text{Representation} \rightarrow \text{Polars}$

This avoids forcing one framework to perform every operation.

---

11. Production Scaling Considerations

The provided package is deliberately simple enough to run locally while preserving the architecture needed for larger workloads. For a much larger data lake, the next step is to move from Single Parquet File to Parquet Dataset or you can say Partitioned Layout data with partition dimensions such as:

date = YYYY-MM-DD/
source = kaggle/
source = ga4/

or a suitable combination based on query workload. The important principle is that the partition key should reflect common access predicates. A partitioning scheme that is never filtered can add complexity without improving scan efficiency. DuckDB also supports glob-based Parquet scans and Hive-style partition discovery, making it practical to scale the same SQL patterns from one file to a dataset layout.

---

12. Reproducibility Contract

A dataset experiment should be uniquely identifiable by:

$\text{ExperimentID} = \text{Hash}(\text{Source Snapshot, Config, Code Version, Random Seed})$

At minimum persist:

• seed
• source paths
• source hashes
• pipeline configuration
• code version
• noise configuration
• row counts
• schema fingerprints

The current pipeline persists the realized noise counts in noise_report.json. For a full production system, that report should become part of a dataset manifest.

---

13. Recommended Next-Step Modeling Contract

The resulting dataset should be consumed by the recommendation stack built in Part 1. The logical chain is:

$$\begin{aligned} \text{Raw Events} &\rightarrow \text{Unified Events} \rightarrow \text{Noise Injection} \\ &\rightarrow \text{Implicit Interactions} \rightarrow \text{Feature Tables} \\ &\rightarrow \text{Candidate Retrieval} \rightarrow \text{Ranking} \end{aligned}$$

The key model-facing artifacts are:

1. Collaborative signal: $R_{ui}$ or confidence-weighted implicit interactions.
2. Item side features $X_i = [\text{category, brand, price, freshness, coldStart}]$
3. User side features: $X_u = [\text{frequency, recency, region, LTV, session depth}]$
4. Context features: $X_{u,i,t} = [\text{recent intent, device, geo, promotion, time}]$

This gives the downstream model a realistic feature hierarchy instead of an unnaturally sparse user-item matrix alone.

---

14. Critical Engineering Caveats

The resulting dataset is intentionally realistic but is still synthetic in important ways. The anomaly distributions are configuration-driven approximations, not claims about the exact failure rates of a particular ZaPudia production system. For example $P(drop)=0.08$, is a controlled experimental parameter.

It should not be interpreted as an empirical estimate of actual GA4 event loss. Likewise, the bot-flood parameter in §2.2 is set so that synthetic bot volume is approximately 5% of the human event volume, a controlled, detectable load chosen to exercise the bot-filter preprocessor, not an empirical estimate of real ZaPudia traffic composition.

It does not mean that five percent of real ZaPudia traffic is bots. The dataset is therefore best understood as:

$\text{Synthetic Data} + \text{Empirical Source Characteristics} + \text{Controlled Failure Modes}$

rather than a literal replica of production.

---

15. Final Data Engineering Blueprint

Zooming back out, the final architecture is the same picture introduced in “Architecture at a Glance” at the top of this article, the loop is now complete, from raw supplier to production system:

The one-sentence business summary: we do not clean the data up before training, we deliberately dirty it in the same ways production will dirty it, so the model’s report card in Part 3 tells us the truth about how it will perform on launch day, not a flattering story about how it performs on a spotless practice exam.

The architectural objective can be summarized as:

$\boxed{P_{train}(X) \approx P_{\text{production}}(X)}$

not because the training dataset is artificially sanitized, but because the training data deliberately contains the failure modes the production pipeline is expected to encounter. That is the central idea behind the Ultimate ZaPudia Unified Dataset. A recommender should not merely learn from what users do when the telemetry pipeline behaves perfectly. It should learn and be evaluated against a world where:

1. events disappear
2. bots fabricate activity
3. timestamps disagree
4. catalog attributes vanish
5. new products have no history

The pipeline in this part converts those realities into deterministic engineering inputs. That provides the data foundation required for Part 3: training, evaluation, retrieval, ranking, leakage prevention, and production model validation on the Ultimate ZaPudia dataset. The GA4 schema discussion in this article follows Google’s documented BigQuery Export schema, including event fields, user_pseudo_id, device fields, and geo fields. The Parquet execution model follows DuckDB’s documented read_parquet and Parquet scan behavior.

---

16. Bridge to Part 4

Part 1 - 2 defined the architecture and mathematics. Part 3 (this article) produced a realistic, telemetry-degraded dataset that reproduces the production failure modes the architecture must survive: telemetry drops, bot floods, out-of-order events, schema degradation, and cold-start rot.

Part 4 will consume these parquet artifacts to:

• Train the iALS, BPR, and Two-Tower models from Part 1 §4 on the corrupted dataset, measuring robustness to each injected pathology.
• Build the LightGBM LambdaMART re-ranker over the engineered features and the implicit matrix.
• Implement the bot-filter preprocessor and the session-reconstruction (out-of-order tolerant) module that the dataset’s flaws demand.
• Run the offline metric framework of Part 1 §5 (NDCG@K, MAP@K, ILD, Gini) to select the final model portfolio per scenario.

The dataset is the contract between data engineering and ML; the noise is the test of whether the contract holds.