SCADA System Audit Using Observations
En la era de la digitalización industrial, la confianza depositada en los datos telemétricos es casi absoluta. Sin embargo, para el Ingeniero de Planta y el…
In the era of industrial digitalization, the trust placed in telemetric data is nearly absolute. Yet for the experienced Plant Engineer and Operations Director, an uncomfortable truth remains: the map is not the territory.
The discrepancy between data reported by SCADA/MES systems and the physical reality of the shop floor is known as the "Digital Reality Gap." This technical article explores how the Work Sampling methodology—grounded in Tippett's technique and statistical inference—serves as the only audit tool capable of validating, calibrating, and correcting the deterministic biases of automated sensor systems without invasive hardware.
1. The "Digital Reality Gap" in Industry 4.0
1.1. The illusion of control in dashboards
The massive deployment of data-acquisition systems creates a sense of omniscience. Dashboards show OEE (Overall Equipment Effectiveness) KPIs in real time, calculated automatically. Software tools such as Induly are vital for continuous OEE management and cost control; however, the precision of these systems depends intrinsically on the quality of the physical input. A green dashboard does not always imply an efficient plant; often it only indicates a plant that knows how to report data to make itself look green.
1.2. Sensor blindness: Deterministic limitations of PLCs and MES
PLCs operate under deterministic binary logic or state registers (e.g., Run, Stop, Fault). They lack causal context. A sensor can detect that a motor has stopped, but it is incapable of discerning whether the cause is a mechanical failure, a lack of material, or an operator's micro-break. This "sensor blindness" forces reliance on human input via HMI panels, introducing subjectivity into a supposedly objective system.
1.3. Critical differences between telemetry and contextual observation
While telemetry answers the "what" and "when", contextual observation answers the "why." Statistical auditing does not seek to replace telemetry but to validate its correlation with reality.
2. Common Pathologies in Automatic Data Collection
2.1. Imputation Bias: Human errors in HMI panels
When a machine goes into a stop state, the MES system asks the operator for a "cause code." Market research indicates that up to 35% of these imputations are wrong. The reasons range from default bias (selecting the first code on the list to clear the screen quickly) to fear of reprisals (logging a technical stop when it was actually an organizational wait).
2.2. Micro-stops and Speed Loss: What polling rate cannot detect
The polling rate of many old or poorly configured SCADA systems may be 1 to 5 seconds. Micro-stops (minor jams, quick adjustments) that last less than the polling interval are invisible to the system—or are erroneously recorded as operation at reduced speed (Speed Loss), distorting the OEE Performance calculation.
2.3. False Positives: Distinguishing "Run" from running empty
A current sensor or cycle counter can indicate a Run state while the machine runs empty (without processing a part). This artificially inflates Availability and burns through the energy margin—a phenomenon that only direct observation or a detailed energy audit can reveal.
2.4. The Inverse Hawthorne Effect
The classic Hawthorne Effect suggests people improve their performance when observed. In digitalized environments, we observe an "Inverse Hawthorne Effect": operators learn the algorithmic triggers of the MES system and adapt their work pace to manipulate the efficiency log—for example, deliberately slowing manual feeding to avoid registering a micro-stop that would require explanation in the HMI.
3. Foundations of Stochastic Audit (Work Sampling)
To combat these biases, we apply the Work Sampling technique, today facilitated by applications such as WorkSamp, which digitize and ensure the mathematical rigor of the process.
3.1. Applying the Tippett technique in digitalized environments
L.H.C. Tippett demonstrated that instantaneous random observations (Snap Readings) of an industrial system converge stochastically to the system's operational reality, provided the sample is representative.
3.2. Mathematical rigor: Binomial distribution and Gaussian curve
Sampling is based on the law of probabilities. The occurrence of an event (e.g., machine stopped) follows a binomial distribution which, for large samples, approximates a normal distribution (Gaussian Curve).
3.3. Sample size calculation (N)
To audit a SCADA system with scientific validity, we must define a Confidence Level ($Z$)—typically 95% ($1.96\sigma$)—and an acceptable Margin of Error ($e$). The formula for determining the number of observations ($N$) is:
$ N = \frac{Z^2 \cdot p \cdot (1-p)}{e^2} $
Where $p$ is the estimated probability of event occurrence (based on SCADA historical data or a pre-sample).
3.4. Margin of error vs. sensor precision
If the SCADA reports 85% availability and our Work Sampling study, with an error of $\pm 3\%$, yields 75% availability, there is a statistically significant discrepancy that invalidates the digital figure.
4. WorkSamp Data-Comparison Methodology
4.1. Designing the MECE Taxonomy
For the audit to be effective, the observation categories must be MECE (Mutually Exclusive, Collectively Exhaustive) and mappable to the PLC states.
- SCADA data: "State: Stop".
- WorkSamp data: Disaggregates "Stop" into {Material Wait, Cleaning, Breakdown, Personal Break, Meeting}.
4.2. Running parallel Snap Readings
During a representative period $T$, random observation rounds are executed. Using WorkSamp on mobile devices guarantees that recording times are exact and random, eliminating observer bias.
4.3. Histogram overlay
At the end of the study, we overlay the frequency histogram of the Digital Dataset (SCADA) with the Statistical Dataset (WorkSamp). Areas of non-overlap reveal pathologies of the data-collection system.
4.4. Isolating real Wrench Time
The system can record that an operator is "logged in" on a machine, but sampling reveals the true Wrench Time (direct, hands-on work), separating it from travel or administrative time that machine sensors cannot detect.
5. Interpreting Deviations in OEE (Overall Equipment Effectiveness)
5.1. Identifying the null hypothesis rejection zone
If the SCADA figure falls outside the confidence interval calculated by sampling, we reject the null hypothesis that "the SCADA system measures correctly." This triggers corrective action: recalibrate sensors or retrain personnel.
5.2. Segregating setup times (Internal vs. External)
Sensors rarely distinguish between internal setup (machine stopped) and external setup (machine running). The visual audit allows these times to be segregated, which is critical for applying SMED methodologies. For an in-depth analysis of these cycle times and specific movements, tools such as Cronometras are the ideal complement for detailed time studies after a deviation is detected.
5.3. Adjusting Availability and Performance
The end result is a "Correction Factor" that should be applied to the SCADA's historical reports to reflect the operational and financial reality of the plant.
6. 2025 Spanish Regulatory Context and Industry 5.0
6.1. Demands of the Industry Law and Strategic Autonomy
The Spanish regulatory framework toward 2025 emphasizes real efficiency and sustainability. Stochastic auditing makes it possible to certify not only productivity but operational energy efficiency, detecting machines that consume without producing (Energy OEE).
6.2. ROI audit for Kit Digital subsidies and Next Generation funds
Justifying European funds requires empirical evidence of improvement. The comparison "Baseline (Sampling)" vs. "Implementation (Digital)" is the strongest proof of ROI before government auditors.
7. Conclusion: Toward Hybrid (Digital + Analog) Calibration
Management based exclusively on sensors introduces an invisible but costly operational risk. Technology does not eliminate the need for observation; it transforms it.
For Plant Engineers and Operations Directors, the recommendation is clear: do not accept SCADA data as dogma. Implement Quarterly Operational Calibration Audits using the WorkSamp methodology. This hybrid approach ensures that your strategic decisions are based on what really happens on the gemba, and not only on what your sensors "believe" is happening.