Combining Model and Test Data for Optimal Determination of Percentiles and Allowables: CVaR Regression Approach, Part I

Abstract

This report makes a more detailed assessment of the CVaR regression method proposed in [3] for determining A-Basis and B-Basis allowables, and quantifying the impact of test data and different analytical models on failure load predictions. Although the method can in principle be applied to any desired quantile, we will focus only on the 10th (B-Basis) in this study. We consider failure data arising from two sources: (1) a controlled environment where data is simulated from different Weibull distributions; (2) a supplied dataset similar to that of [3] augmented with failure load predictions from two additional analytical models (Model S2 and Model S3). Using absolute deviation between true and estimated 10th percentiles and the CVaR regression goodness-of-fit measure introduced in [3] as accuracy-assessment criteria, the key findings are as follows. The accuracy of CVaR regression is relatively insensitive to the number of batches present, but fairly sensitive to the number of test points per batch 3. There are diminishing benefits in using more than 10 batches, or more than 10 test points per batch, in any one application of CVaR regression. The estimates of A-basis and B-basis are fairly robust, in the sense that they are not severely affected by miscalibrations (biases or errors) in the analytical models. Among the analytical models used as the sole input with no (input) test data, the best performer is Model S, followed by Model S2. Model S3 is the worst performer. The models contribute substantially to percentile prediction when up to 3 test points are used as input. When 4 test points are used as input, the 3 models can be roughly equated to the input information provided by one additional test point. Executive Summary In [3], we proposed a coherent methodology for integrating various sources of variability on properties of materials in order to accurately predict percentiles of their failure load distribution. The approach, CVaR regression, involved the linear combination of factors that are associated with failure load, into a statistical regression or factor model. The method can be used for determining A-Basis and B-Basis allowables, and quantifying the impact of experimental (or test) data and different analytical models on failure load predictions. The present report builds on this work by considering failure data arising from two sources: (1) a controlled environment where data is simulated from different Weibull distributions with 3 A "batch" is defined as any one single source of variability affecting the test point failure data. 192 Stan Uryasev and A. Alexandre Trindade parameters in ranges plausibly mimicking failure data from a supplied dataset similar to that of [3]; (2) a supplied dataset with failure load predictions from three analytical models. Specific Objectives 1. Pooling of various sources of information of possibly different origins: models, experiments and expert opinions. 2. Develop a methodology for estimating percentiles of failure distributions and allowables. 3. Minimize amount of data needed for certification process. 4. Take into account various sources of uncertainty. 5. Validate the approach in a controlled statistical environment. 6. Demonstrate the efficacy of the approach with case studies. Summary of Accomplished Tasks and Findings 1. Developed factor model for direct estimation of percentiles using: a) various sources of information (models, experiments, expert opinions); it is possible to quantify value of different sources of information. b) statistical characteristics: mean, st.dev., deviation CVaR, etc. 2. Simple, clear, computationally effective methodology enables the pooling of data across: a) many individual materials: relatively small requirements on size of datasets. b) various experimental setups: crediting simple experiments to more sophisticated (ex-pensive) ones. 3. Developed CVaR statistical techniques for optimal estimation (weighting) of coefficients in the factor model, and corresponding confidence intervals for unknown parameters (A-basis and B-basis). a) Technique is new; developed in the framework of the AIM-C project. b) Approach was especially designed for estimating percentiles and constructing confidence intervals (A and B-basis). c) No distributional assumptions (such as normality) are made; method is nonparamet-ric. d) CVaR deviation goodness-of-fit measure has exceptional mathematical and computational properties, allowing easy and efficient implementation of the methodology via linear programming: high speed of calculations; analysis of large datasets feasible ; stable results. 4. Case studies were performed with simulated data. a) Minimal number of batches needed to calibrate the CVaR regression model was determined ; sensitivity to this number. b) Minimal number of experimental test points per batch was determined; sensitivity to this number. c) Approximate relationship established between CVaR deviation error (observed) and true error (unobserved) in percentile estimation. d) Sensitivity of the approach to errors in analytical model information was assessed; methodology is robust to biases. e) 10th percentile estimates based on Model S individually, and on Model S plus 5 test points, are close to true values. B-basis values are also close to nominal values based on actual experiments. 5. Two case studies carried out for open-hole coupon dataset: estimation of 10th percentile and B-basis (failure data plus 3 analytical models). a) CVaR regression calculations provided plausible estimates of percentiles of failure load distribution. b) CVaR regression with analytical models only as predictor variables, provide plausible percentile and B-basis estimates, even in the absence of any experimental test data (used as predictors). c) Benefits of combining models for predicting percentiles were quantified. d) Benefits of combining models and experimental data were evaluated. 6. Our investigations provide compelling evidence that the methodology can effectively integrate modeling and experimental data, and reduce overall testing cost.