notation:To: Those concerned with GALP regulations From: William M. Knospler Date: December 19, 1994 Subject:Software Verification and Validation (Preliminary draft)
As the pharmaceutical and chemical research and testing industries phase in and start to comply with GALP, we are becoming more frequently summoned, to provide information and documentation which may help satisfy the reporting and compliance requirements.
This document seeks to provide a general outline regarding the manner and method of software development employed for the purpose of insuring an organized and reliable means for maintaining and improving the product.
The verification of results calculations is treated in considerable depth, since this topic, (the answers), represents the primary reason for using a computer based data acquisition and reduction system in the first place.
Scintflow is the companion software product of the B-RAM radioactivity flow monitor manufactured, sold and serviced by IN/US Systems Inc. The first release, version 1.xx began shipping in late 1988. During 1989 and 1990 the software was re-organized, modernized and re-structured to take advantage of new and improved detector hardware, video displays, computer operating systems and compiler features. The software's underlying structure and foundation have effectively endured the 'test of time' with over 600 B-RAM systems, in some 20 countries, installed around the world.
Improvements to Scintflow since 1990, have been, and continue to be, of an incremental and evolutionary nature, and are incorporated with more rigorous attention to documentation and protocol. The following outline represents the process by which a software feature or improvement is incorporated into the product.
A description of the functional requirements and specifications of the improvement is defined and expanded from the conceptual material, and is documented as a Software Advisory Bulletin, by the software developer. The Software Advisory Bulletin contains the following information, all as applicable:
A program structure analysis is conducted to determine the effects of the improvement:
During the coding and implementation phase organization issues are observed:
The developer tests and verifies that:
Minor changes and cautionary notes are appended to the Software Advisory Bulletins, so they more accurately depict proper and useful operation and advise the user of issues requiring special consideration.
The distributor receives the program version package with the applicable Software Advisory Bulletins, then tests and verifies the system for proper operation, and when satisfied that everything is in order, prepares the version for final release.
User Documentation:
The Software Advisory Bulletins serve as the raw material from which the distributor produces the necessary user documentation and incorporates it into the instruction manual. The Software Advisory Bulletins are normally included in an appendix of the instruction manual. The distributor and the developer provide recommendations and guidelines which a user may find useful when developing SOPs for a particular study.
An Anomaly Report form is provided for the purpose of reporting a software problem.
This section addresses the important issue of results calculations. A user should 'know' how a result will be calculated, and agree that calculations carried out by the software are in substantial agreement with those commonly practiced in the industry. The dynamics of a flow system require that some attention be paid to details, variable background for example, which are not an issue in a static sample counter.
A data file named 'STEPCAL', (included in the Scintflow distribution package), provides a means of verifying the correctness of the calculations which Scintflow performs during the normal course of system use. A user, concerned with calculations, or one who needs to validate system output, in order to comply with QA, QC or other regulatory requirements, can use STEPCAL as a part of that effort. Further, STEPCAL can be useful as a training aid, when factors affecting the results output need to be taught or clarified.
TDP -- Time per Data Point -- Sometimes called the Dwell Period, it is the amount of time which elapses between successive data points and is usually set at from 1 to 6 sec.
CPM -- Counts Per Minute - - The number of decay events which would be detected if a sample were counted for a full minute.
Obs -- Observed counts -- Actual counts detected by the radioactivity detector.
FF --Flow Factor -- The composite factor calculated from the flow rate, cell volume
and (if liquid scintillator is used, the effect of sample dilution).
FF = Fa + (Fa * R) / Vc where
Fa = Flow rate of analyte in ml/min
R = Ratio of liquid Scintillator to Analyte.
Omit (Fa * R) when Liquid Scintillator is not used.
Vc = Cell Volume in ml
FCD -- Flow Corrected Data -- Results with the FF applied
Eff -- Detector Efficiency -- in %. 100 % forces the results to be reported in CPM.
Bkg -- Background level -- The level of radioactivity regarded as the baseline, on top of which the sample's radioactivity level appears. The value is given in CPM, and is subtracted from the Observed counts before the Flow Factor and Efficiency are applied.
LTL -- Lower Time Limit -- of integration
UTL -- Upper Time Limit -- of integration
ROI -- Region Of Interest -- A segment of the data trace bound by the data above, the X- axis below, the LTL on the left and the UTL -1 on the right, and which usually has some significance as a result, to the user
notation:It is customary to refer to peak areas of radioactivity in CPM as integrals, as though we were evaluating a continuous mathematical function, f(x) over limits of integration, which of course we are not.
Radioactive decay events are discrete, and they are counted for discrete periods of time (TDP). An analog of the number of observed counts in a given time period, is not needed (except to possibly
transfer the information to some other data collection system's Analog Input); it
is truly a digital representation, not requiring calibration or conversion. The number
of decay events observed (Obs) during a single time period (TDP) is stored as an integer
and represents one data point on a computer display.
So when we speak of integration we are really referring to the summation of discrete events and we sometimes even use the correct (sigma) notation, . We use the term integral as a convention to denote
the area under the curve in CPM, but do not imply a mathematical expression by it.
When we select an ROI and perform a summation, the resulting CPM value is equivalent to collecting the sample stream from the LTL to the UTL, then counting it in a static sample counter, which too can report the radioactivity level in CPM.
The CPM reported for any ROI is calculated as follows:
T (in minutes) = (UTL - LTL) * TDP / 60
Obs (net) = - (Bkg CPM * T) CPM = Obs (net) * FF
DPM = CPM / Eff * 100
Note: Bkg (CPM * T) is subtracted from summed Obs, yielding Obs (net counts). Restating, a minutes worth of Background is subtracted from a minutes worth of Obs in an ROI, proportionally.
The Cursor is used to tag the lower and upper limits of ROIs. CPM Integrals can be observed immediately by placing the cursor anywhere within the ROI and reading the result in the Cursor information block located immediately to the left of the trace's Y-axis. When tagging ROIs, the following cursor operation should be noted.
The "B" (for Background) field of the Trace Information Block will be referred to when dealing with Background subtraction. It is shown at the third line from the top. To enter a declared Background value, click the left mouse button when pointing to any portion of the "B" or the value beside it. A value entry field will be highlighted and receive user input.
An explanation of STEPCAL's use will be limited to Radioactivity Trace 1 only, however the same instruction applies to any trace. The discussion here assumes that the reader has previously acquired experience and understanding of the functions and features of the Scintflow software package in the Evaluation mode. In the event that some further reference to subject matter is indicated, the relevant portion of the user manual can be viewed on screen by highlighting the function and executing <Alt+F1>.
Acquisition parameters of STEPCAL affecting the results calculations are as follows
The flow rates and cell volume produce a Flow Factor of 1.00, so observed (Obs) and flow corrected data (FCD) will be the same.
Each trace is composed of a series of steps of known values. Each step represents a constant activity level for the duration of the step. The following table lists the step values and their times.
Table 1. The table lists the start and stop times, inclusive for which the data values are constant.
Table 2. Shows the LTL and UTL for tagged regions which include the entire step. As mentioned, the data point corresponding to UTL of each region does not contribute its value to the region's integral.
Fig. 1 Initial conditions of STEPCAL.R01
The 600 CPM rectangle is the basic unit of measure when evaluating STEPCAL. The whole minute marks are extended vertically to intersect the steps, and the steps are extended horizontally, to intersect the 10 minute mark. A group of rectangles is formed, each 10 counts high and 1 min (60 data points at 1 sec each) long. The area of each rectangle is 600 CPM; one half rectangle = 300 CPM; one and one half rectangle = 900 CPM, etc. A total of 20 rectangles fit within the entire trace, so the total area of the trace (with Bkg = 0) is 20 * 600 CPM, or 12,000 CPM.
Fig. 2 - The 600 CPM Rectangle
Region Name Start Stop Reten Sum %SPks %STotal ----------------------------------------------------------------------------------------- 1 - 2:00 3:00 2:00 600 5.00 5.00 2 - 3:00 4:00 3:00 600 5.00 5.00 3 - 4:00 5:00 4:00 1200 10.00 10.00 4 - 5:00 6:00 5:00 1200 10.00 10.00 5 - 6:00 7:00 6:00 1800 15.00 15.00 6 - 7:00 8:00 7:00 1800 15.00 15.00 7 - 8:00 9:00 8:00 2400 20.00 20.00 8 - 9:00 10:00 9:00 2400 20.00 20.00 ---------------------------------------------------------------------------------------- Sum all H-3 (1) = 12000 CPM (Net) Background removed = 0 Counts Sum all tagged peaks = 12000 CPM (100.00%)
Note: The retention time reported for normal peaks is the time of peak maxima. Since no data point within these regions was greater than the first point, its time is reported as the retention time.
Fig. 3 - The two minute Region
Region Name Start Stop Reten Sum %SPks. %STotal --------------------------------------------------------------------------------------- 1 - 2:00 4:00 2:00 1200 10.00 10.00 2 - 4:00 6:00 4:00 2400 20.00 20.00 3 - 6:00 8:00 6:00 3600 30.00 30.00 4 - 8:00 10:00 8:00 4800 40.00 40.00 --------------------------------------------------------------------------------------- Sum all H-3 (1) = 12000 CPM (Net) Background removed = 0 Counts Sum all tagged peaks = 12000 CPM (100.00%)
*Note: When the display is set to narrow, there are 430 horizontal screen pixels between the Y-axis and the beginning of the trace information block. There are 600 data points in a 10 minute chromatogram at 1 sec / data point. Mapping 600 data points onto 430 pixels requires that 170 data points be omitted from the display, which forms compression, and the 30 second points of the chromatogram were among them. With 8 min of data, 480 data points are mapped onto 430 pixels, with only 50 data points omitted from the display. and the 30 second points are not among them. When there are fewer data points to be displayed than there are pixels the situation is reversed, and some points will occupy more than one pixel to form expansion. When the display is set wide, there are 530 horizontal pixels available, but the same principals still apply but with different numbers. In any case all data points within an ROI are included in the calculations whether displayed or not.
The form of Background processing to be applied to the results is selected by the user and is dependent on his or her preferences and / or operational or administrative requirements. To graphically view the data level representing the Background, select Baseline from the HotKey list. When a check mark ' ' appears next to the Baseline selection, Background display is active. The results table's Integral column may indicate negative numbers to signal the user of an ambiguity (the Background subtracted is greater than the available Observed counts in the ROI). The %Pks. column only includes regions with ROIs whose areas greater than zero.
The Background processing options adhere to an order of precedence. The following Background processing options are available:
Each method of processing background will be fully described.
Fig. 4
Table 4.1 Declared background is 0 CPM
Region Name Start Stop Reten. Sum %SPks. %STotal ------------------------------------------------------------------ 1 - 1:00 2:00 1:00 0 0.00 0.00 2 * - 2:00 2:10 2:00 100 1.14 0.83 3 * - 3:00 3:20 3:00 200 2.29 1.67 4 * - 4:00 4:40 4:00 800 9.14 6.67 5 * - 5:00 5:50 5:00 1000 11.43 8.33 6 - 6:00 7:15 6:00 2250 25.71 18.75 7 - 8:00 9:50 8:00 4400 50.29 36.67 ------------------------------------------------------------------- Sum all H-3 (1) = 12000 CPM (Net) Background removed = 0 Counts Sum all tagged peaks = 8750 CPM (72.92%)
Table 4.2 Declared background is 300 CPM
Region Name Start Stop Reten. Sum %SPks. %STotal ------------------------------------------------------------------ 1 - 1:00 2:00 1:00 -300 0.00 0.00 2 * - 2:00 2:10 2:00 50 0.69 0.52 3 * - 3:00 3:20 3:00 100 1.38 1.04 4 * - 4:00 4:40 4:00 600 8.30 6.26 5 * - 5:00 5:50 5:00 750 10.38 7.82 6 - 6:00 7:15 6:00 1875 25.95 19.55 7 - 8:00 9:50 8:00 3850 53.29 40.15 ------------------------------------------------------------------ Sum all H-3 (1) = 9590 CPM (Net) Background removed = 2410 Counts Sum all tagged peaks = 7225 CPM (75.34%)
Table 4.3
Fig. 5 A Background region is tagged on the 600 CPM step and reports a Background value of 600 CPM
Fig. 6 A Background region is tagged half on the 0 CPM step and half on the 600 CPM step and reports a Background value of 300 CPM.
Fig. 7 There are four 600 CPM rectangles totaling 2400 CPM in a Background region spanning 4 min. equaling an average Background value of 600 CPM.
Fig. 8
Region Name Start Stop Reten. Sum %SPks. %STotal ----------------------------------------------------------------- 1 - 1:59 3:59 2:00 590 25.00 4.92 2 - 3:59 5:59 4:00 590 25.00 4.92 3 - 5:59 7:59 6:00 590 25.00 4.92 4 - 7:59 9:59 8:00 590 25.00 4.92 ----------------------------------------------------------------- Sum all H-3 (1) = 12000 CPM (Net) Background removed = 0 Counts Sum all tagged peaks = 2360 CPM ( 19.67% )
Fig. 9
Region Name Start Stop Reten. Sum %SPks. %STotal ------------------------------------------------------------------ 1 - 2:00 3:00 2:00 0 0.00 0.00 2 - 3:00 4:00 3:00 -300 0.00 0.00 3 - 4:00 5:00 4:00 0 0.00 0.00 4 - 5:00 6:00 5:00 -300 0.00 0.00 5 - 6:00 7:00 6:00 0 0.00 0.00 6 - 7:00 8:00 7:00 -300 0.00 0.00 7 - 8:00 9:00 8:00 0 0.00 0.00 ------------------------------------------------------------------ Sum all H-3 (1) = 12000 CPM (Net) Background removed = 0 Counts Sum all tagged peaks = 0 CPM (0.00%)
Since decay events are discreet and counted directly, no calibration or conversion of data values from one form to another is necessary. The only concern regarding the process of acquiring the counts and expressing them in CPM, is the accuracy and stability of the time base itself. The B-RAM hardware employs a precision crystal oscillator which is divided by an integer value producing exactly a 100 millisecond time base. Decay events are accumulated in hardware for 10 timer ticks. The events are then summed for the period specified by the TDP and transferred to the software to become a single data point on the trace. Scintflow software plays no part in establishing or controlling the time base. The maximum time jitter introduced by B-RAM's CPU interrupt latency in one second, is approx. .00015%, and the manufacturing tolerance of the crystal itself is typically less than .05% , making the total worst case time base error equal or about .0501% or 501 PPM. The following experiment tests for systematic error by generating a count rate histogram of an actual radioactive source, and comparing the results to a mathematical model.
Fig. 10
------------------------------------------------------------------ Trace 1 Chan 1 Histogram Data Factors are Applied! Baseline1 0.00 Baseline2 0.00 ConvFact UserFact 1 ------------------------------------------------------------------ Reg Name Range Mean StdDev #Events %SReg %SAll <------ct/.1sec -----> ------------------------------------------------------------------ 1 __________ 35 - 91 60.6 7.9 23415 100.00 99.96 ----------------------------------------------------------------- All Chan1 = 23424 Events 23415 100.00 99.96
Fig. 11
------------------------------------------------------------------ Trace 1 Chan 1 Histogram Data Factors are Applied! Baseline1 0.00 Baseline2 0.00 ConvFact UserFact 1 ------------------------------------------------------------------ Reg Name Range Mean StdDev #Events %SReg %SAll <-----ct/.1sec ------> ----------------------------------------------------------------- 1 __________ 37 - 45 42.2 1.8 91 1.67 1.67 2 __________ 45 - 53 49.5 2.1 3224 13.81 13.76 3 __________ 53 - 61 56.8 2.2 8188 35.06 34.96 4 __________ 61 - 69 4.2 2.3 7901 33.83 33.73 5 __________ 69 - 77 71.6 2.2 3108 13.31 13.27 6 __________ 77 - 85 9.3 2.1 540 2.31 2.31 ------------------------------------------------------------------ All Chan1 = 23424 Events 23352 100.00 99.69
In the process of software validation, it is likely that some form of computer output, either graphic or tabular, should accompany the software validation documentation. In that regard, one may find the methods of capturing and importing DOS's graphics screens into a document useful.