How to compare complex systems is always a challenge. Recently TechniCom has been using a technique called Delphi Expert Analysis. The idea being that asking a question of an experienced user should result in a cogent response of how well he perceives a system performs a specific task. The results depend on asking the proper questions, selecting truly expert users, and managing the process.

Our experience using TechniCom’s Delphi Expert Analysis shows that this is a solid way to perform market research on technical software. Instead of a simple feature and function analysis, the Delphi expert approach relies on the opinions of independent external expert users who rate and comment on a series of questions prepared by TechniCom. Developing the most appropriate questions ensures that our analysis is correct. We recruit a number of experts for each system being evaluated. Ideally each group of experts has correspondingly similar backgrounds. We provide a series of detailed questions to each expert, closely monitoring their progress and working with them to insure similar levels of evaluation to those of the other experts. The detailed questions are each rated subjectively for each system along with each expert providing comment that justifies each rating. For instance, if the expert rated that the question scored a 2, then the comment explains why. The scores range from 0 to 5, with 5 being the best, and 0 depicting no capability. A rating of 5 could be considered perfect — the question being evaluated meets all requirements and can expand to meet future needs. Very few 5’s are awarded. Out of range ratings are explored individually with the expert.

Each expert gets a final pass at their evaluation after receiving a report showing the results of their peers. This proves highly effective in normalizing the results.

We then accumulate all answers and summarize the results.

On Monday, 9 Aug 2010, a summary of our most recent analysis using this technique to compare Autodesk Inventor and SolidWorks will be posted to http://www.cad-portal.com .