Don,

Gorur and I are debating this in an email right now. I will post the entire thread when we have come to a conclusion. It's a bit more complex than I thought.

Check back in a day or so.

Hi Don, Gorur and I went back and forth on this. Not sure we came to an agreement, but the discussion might be worth reading.

Gorur Sridhar:
It is a good question and I am glad to note that other applications of the 6 Sigma tools are being thought of. Since contract review is a one time activity and is not measured against any requirement or service level agreement (SLA) there is no scope for either repeatability nor reproducibility. I don't think R&R study would work in that case.

Dirk Dusharme:
I'm not sure I agree. It might depend on the contract review. My thoughts are that there are many places for variability in the process.

Contract itself:
Like a gauge, a contract can be well designed, or not. A properly written contract will be easy for a person to understand and fill out. A poorly written one will cause the user to not fill out properly, or to have to continuously keep asking questions about this or that on the contract.

Contract review:
As with your engineering drawing checklist, there can be a lot of variability. Is the contract reviewer checking everything he should be checking? Does she understand what she should be checking, and so forth. Are the checks themselves being done properly (perhaps the reviewer has to run a credit check on the signer. Are they doing that properly).

I would think a gauge R&R study would show up the same discrepancies as in your example. Don't you think?

GS:
Contracts will vary from project to project and from customer to customer, depending on how involved the project is. Especially in an engineering service industry many times the customers will not bother about the contract reviews but are OK if the job is well done. Also, due to the above reason, there cannot be any fixed template designed to capture all the required points. In order to get all the requirements captured, the contracts will be followed up by "query clarification" logs which supplement the requirements, and in cases where the response is not forthcoming, then an "assumption" is made and conveyed to the customer as FYI.

Due to the above reasons the contract review can never be considered as "right" or "wrong" or "complete" or "incomplete". Unlike in gage R&R wherein you measure against a fixed value which you ought to get no matter the number of times you measure, contract review cannot be done so, and also unlike gage R&R there is no reproducibility or repeatability as the contract is given once (no reproducibility) and checked only once (no repeatability) because normally the team size in a marketing dept. would be only 1 or 2 persons and they would be focusing on different types of customers without overlap.

DD:
I think I see why we might be disagreeing. When I am talking about contract review, I am thinking about a contract in general terms, not necessarily engineering services.

I think you are talking about contract review for the engineering services company in your example. Am I right?

My thinking is that for, let's say, a contract for a life insurance policy. The contract is a standard contract for that particular type of insurance, with a lot of blanks for the customer to fill in. That contract is reviewed by the insurance underwriter. Might not GRR work in that case?

GS:
Yes, I was referring to eng. services.

A life insurance policy or medical insurance or vehicle etc... will not have the R&R element in it. It will not be filled out twice to get the reproducibility aspect nor reviewed twice for repeatability. Leaving out blanks does not mean variability, it can be for not being aware of. Variability is doing the right thing but not achieving the desired results... right??

DD:
Ok. I see your point on the filling out the form. But couldn't an R&R point out that the same "types" of mistakes are made when the form is filled out.

For instance, for whatever reason, people put both their first and last name in the FIRST NAME blank instead of putting their first name in the FIRST NAME blank and last name in the LAST NAME blank. That would point to the form being confusing to understand.

In other words, even though you can't control who is filling out a form, you can spot particular types of errors. Typically, when the same type of error is spotted over and over again, it is an indication that a form is confusing, and the variability (although coming from the people filling out the form) is truly due to the form itself being of poor quality. Couldn't a GRR study examine the types of errors found in two different versions of the form to see which form is better?

For instance:
Form "A" is filled out by 100 random customers. Types/quantities of errors noted
Form "A" is reviewed multiple times by multiple contract reviewers using a check list (similar to your reviewers in your example)
Form "B" is filled out by 100 random customers. Types/quantities of errors noted
Form "B" is reviewed multiple times by multiple contract reviewers using a check list (similar to your reviewers in your example)

GS:
I think you are trying to make the form more robust and user friendly to use. To do this you need not spend effort in reviewing it multiple times, rather a survey sent out will reveal how simple or complicated the form is. Survey will make the job much simpler.
Also getting the forms A and B by 100 random customers each will again introduce variability. It has to be by the same set of people.

Also the stakeholders involved in R&R would have to have a say in the activity and corrective actions can be taken viz. teaching them as to how to perform better, but if outsiders are involved then there is no closed loop feedback mechanism for corrective actions to be put in place.

Don't you think a simple questionnaire and/or a survey will serve the purpose?

DD:
Let's go back to the root of our hypothetical problem, the variability of the contract reviewers.

Couldn't it perhaps work this way:
Ignore the contract and how it was filled out (we are assuming a standard contract is sent to everyone). Let's just focus on the actual errors in the contract review process itself.

In our example of a contract, the part (the contract) never changes.

We want to find out if our reviewers are missing things from the gauge (the checklist) just as you did in your example.

If you ran the same experiment that you ran in your example, where multiple reviewers each examine the same group of sample contracts to their checklist, wouldn't you be checking the variability in the reviewers and the checklist. The sample would be the same, but it would be checked by different reviewers (variability between reviewers) and the samples would be checked multiple times by the same reviewer (variability within a reviewer).

GS:
You are talking and imagining a hypothetical situation where you would have surplus manpower to carry out reviews. But I am talking from the practical point of view of an industry wherein you have the optimum resources with manpower, time etc. Else gage R&R can be used for anything that is being done. This is not feasible. We have to see both from the practical and the theoretical point of views.

DD:
Agreed. It would take manpower. It was more an exercise to see if what I suggested in my last email made sense. I appreciate you taking the time to advance my knowledge on this.

But to get to I what I think was the reader's actual problem. I am assuming that his company is having problems with inconsistent contract reviews. In other words, one reviewer might find no problem with a contract, but another reviewer would (as with your engineers and the checklist). So there is variability in how his people review contracts. So the question (as with your example) is whether the problem is with the checklist or the reviewers or both. This is assuming that the contracts are standard. No difference from one contract to the next other that the information that is filled in.

GS:
I think R&R can be theoretically adopted to any situation provided we have the requisite resources. Else if the issue is inconsistency in filling out forms, then educating the customer by a survey etc. will help and would be an easier solution.