The Pilot Study: Refining the Sample

What is a pilot study in Work Sampling and why is it the most critical phase?

Imagine you are about to diagnose the productivity of a 200-operator plant. You have the statistical formula, the resources, and the schedule. But what if your activity categorization does not reflect operational reality? What if process variability is much greater than you assumed? This is where the pilot study stops being a "formality" and becomes the empirical validation mechanism that separates a robust diagnosis from a theoretical exercise.

The pilot is not simply "doing a few test observations." It is the phase where we compare the theoretical sample size (calculated N) with the actual sample size needed (operational N). Data from our technical report are blunt: projects that skip this phase show 34% more variability in their final estimates.

Skipping the pilot carries two serious consequences:

Massive oversampling: Assuming the worst-case statistic (p=0.5) can lead you to collect 40% to 60% more observations than necessary, inflating costs and timelines.
Loss of credibility: A diagnosis based on an undersized sample or flawed categories does not withstand middle-management scrutiny—invalidating the whole effort.

The statistical problem: When the sample size formula fails in practice

The canonical sampling formula for proportions, N = (Z² × p × (1 - p)) / E², is elegant in theory. Its Achilles' heel is the variable p: the population proportion of the activity you want to measure (for example, "wrench time" or net productive time).

Before observing, p is unknown. The conservative practice is to assume p = 0.5, the value that maximizes the product p × (1-p) and therefore the sample size. This is the "statistical worst case."

In real industrial environments, this assumption is almost always wrong and costly. If the true proportion of productive activity is 30% (p=0.3), the calculation with p=0.5 will give you a much larger N than necessary. The pilot solves this problem by providing an initial estimate of p (p̂) based on empirical data, allowing you to recalculate N precisely.

Tippett's technique applied to the 21st century: Progressive estimation of the population proportion

The solution to this dilemma is not new. L.H.C. Tippett described it in the 1930s in the textile industry: iterative and progressive estimation of the population proportion. The pilot study is the modern application of this principle.

As you accumulate observations in the pilot, the estimated proportion p̂ begins to converge toward the true proportion p. This convergence path follows a normal distribution (Gaussian Curve), and its initial standard error gives us a measure of uncertainty.

The pilot captures the first and crucial iterations of this curve. Its value lies not in the final number of observations but in the convergence path it reveals. It lets us see whether our estimate stabilizes quickly or shows high volatility—indicating a greater need for sampling. Tools like Cronometras make it possible to record and visualize this convergence in real time, turning a statistical concept into an understandable management chart.

Validating the MECE taxonomy: The pilot as a categorisation-failure detector

Before launching the full study, we design an activity taxonomy (setup, operation, waiting, transport, etc.). We assume it is MECE: Mutually Exclusive and Collectively Exhaustive. That is, every observed activity fits into one—and only one—category, and together all categories cover 100% of what the operator does.

The pilot is the crucible where this assumption is melted and tested. Our research in Spanish plants reveals systematic failures:

Overlapping categories (67% of pilots): An operator "cleans the workstation" while "supervises an automatic machine." Which category dominates? This ambiguity, if unresolved, contaminates the data.
Unanticipated activities (48% of pilots): Unforeseen tasks always appear, such as "helping a colleague in the adjacent cell" or "finding a supervisor for a decision."
Very low-frequency activities (<5%) (81% of pilots): They require a differential sampling approach or a calculated increase in the total N to capture them with statistical significance.

The pilot forces you to refine the taxonomy before it is too late, saving rework and avoiding the classic "Other" catch-all bucket.

The Hawthorne Effect in the pilot phase: How to measure and compensate for the human factor

The Hawthorne Effect is the change in worker behavior caused by the fact of being observed. In our experience, it is measurable in 39% of pilots run in Spain. During the pilot, productive activity may increase artificially.

Ignoring this effect leads to overestimating real "wrench time." The pilot, precisely because of its short duration and "test" nature, allows us to quantify this bias. A common strategy is to compare observations from the first half of the pilot with those from the second, when workers have partially "gotten used to" the observer's presence.

For the main phase, temporal compensation strategies are implemented: slightly extending the observation period or applying a correction factor based on the deviation detected during the pilot. The goal is to isolate "natural behavior" from "reactive behavior."

2025 Spanish industrial context: Variables the pilot must capture

Spanish industry has particularities that the pilot must absorb for the main study to be valid:

Fragmented 12-hour shifts: In sectors such as chemicals or food, the pilot must cover at least one full shift cycle to capture variations in pace, fatigue, and scheduled stops.
High operator polyvalence: 41% of operators in medium-sized plants perform multiple functions. The pilot helps define whether sampling is done by station or by operator, and adjusts the MECE taxonomy for multi-role functions.
Partial automation: In semi-automatic environments, "wrench time" blends with system-supervision tasks. The pilot is crucial for defining the boundaries between "operating," "supervising," and "intervening"—key concepts for calculating an OEE without sensors based on direct observation.

Production-control platforms such as Induly can later cross-reference sampling data with machine-stop records, validating the consistency of the observation-based diagnosis.

Spanish regulatory framework for work observation

Any direct-observation study in Spain must fit within rigorous legal compliance. The pilot is the ideal moment to test and adjust compliance protocols.

Data Protection (GDPR and LO 3/2018): Observation through "snap readings" is not continuous surveillance, but it does require prior communication to workers' legal representation (Works Council or delegates). The pilot allows this communication to be formalized and clarifies that no identifying personal data is captured—only activity categories.
Occupational Risk Prevention (Law 31/1995): The observer must be included in the risk assessment of the areas they visit. In the pilot, high-hazard zones are identified (confined spaces, proximity to machinery) and specific safety protocols for the main phase are established.
Negotiation with union representation: In plants with an active Works Council, the pilot becomes a transparency tool. It makes it possible to show the anonymous, statistical nature of the method, reducing resistance (present in 30% of initial rollouts) and agreeing on observation protocols accepted by all.

Conclusion: The pilot as an investment in precision, not an expense

The pilot study in Work Sampling is the process by which we transfer the method from the theoretical world to the real one. It is the phase where we detect flaws in categorization, adjust the sample size with empirical data, quantify the human factor, and validate regulatory compliance.

Skipping it is not a saving; it is a high-risk bet that compromises the scientific validity and management acceptance of the final diagnosis. In the data era—where we seek to calculate indicators like OEE without costly sensors—the methodological rigor of work sampling, initiated with a solid pilot, is more relevant than ever. It is the foundation that allows methods-engineering and productivity-improvement decisions to be made with confidence.

Resources and Tools

To delve deeper into time-study and sampling methodologies, explore the ASETEMYT Directory, the most complete catalog of tools and specialists in the sector.
Read more articles on methods engineering and productivity on the ASETEMYT Blog.
If you offer services or software related to industrial timekeeping, you can list your company in the directory.
WorkSamp: Specialists in Work Sampling for productivity diagnosis.
Cronometras: Software for time-and-motion analysis, complementary to sampling studies.
Induly: Production Control and Industrial Time-Clock software to monitor in real time what sampling diagnoses.