Human and Machine Reliability in Postural Assessment of Forest Operations by OWAS Method: Level of Agreement and Time Resources

Abstract

In forest operations, traditional ergonomic studies have been carried out by assessing body posture manually, but such assessments may suffer in terms of efficiency and reliability. Advancements in machine learning provided the opportunity to overcome many of the limitations of the manual approach. This study evaluated the intra- and inter-reliability of postural assessments in manual and motor-manual forest operations using the Ovako Working Posture Analysing System (OWAS)—which is one of the most used methods in forest operations ergonomics—by considering the predictions of a deep learning model as reference data and the rating inputs of three raters done in two replicates, over 100 images. The results indicated moderate to almost perfect intra-rater agreement (Cohen’s kappa = 0.48–1.00) and slight to substantial agreement (Cohen’s kappa = 0.02–0.64) among human raters. Inter-rater agreement between pairwise human-model datasets ranged from poor to fair (Cohen’s kappa = −0.03–0.34) and from fair to moderate when integrating all the human ratings with those of the model (Fleiss’ kappa = 0.28–0.49). The deep learning (DL) model highly outperformed human raters in assessment speed, requiring just one second per image, which, on average, was 19 to 53 times faster compared to human ratings. These findings highlight the efficiency and potential of integrating DL algorithms into OWAS assessments, offering a rapid and resource-efficient alternative while maintaining comparable reliability. However, challenges remain regarding subjective interpretations of complex postures. Future research should focus on refining algorithm parameters, enhancing human rater training, and expanding annotated datasets to improve alignment between model outputs and human assessments, advancing postural assessments in forest operations.