35 ES Development process-II

Bhushan Trivedi

Introduction

We have seen first two steps of development of ES in the previous module. In this module we will see the rest 4. We will begin with prototype construction and end with testing. We will see a process of conceptualization (which is similar to analysis) and formalization (which is similar to design) followed by implementation. Once the implementation planning is done, testing begins. The testing is done for both, functioning for what is expected for the ES to do and, for future evolution. We will begin with prototype construction and conceptualization process.

Prototype Construction and Conceptualization

The second phase of ES development process begins with prototype construction and conceptualization. Conceptualization is modelled as prototype so both things happen in parallel. The prototype construction in the initial run exposes the initial conceptual clarity of the stakeholders and final prototype construction exhibits the conceptual model prevails in their mind. The prototype contains some important part of the system and clearly conveys what the final system will look like. The prototype can present a very simplistic view or can be almost as detailed as the final system. The idea is to start with the minimum requirement and receive experts’ and normal user’s feedback and recursively improve it based on the feedback 1. In extreme cases it is quite possible that the complete prototype may be discarded. Even in that case, it throws clear light to the developer what is lacking and how to go about the next version. Thus the ES usually is iteratively developed by continuously modifying the prototype to converge into a final system. This approach is prefered as nobody is usually aware about what the system should contain in the beginning and that understanding develops gradually. Prototype construction process helps that.

The prototype construction process is quite analogous with drawing a sketch of a building before actually constructing it. The sketch clearly indicates different parts of the building and how they look like. Also, it is much easier to change the sketch before construction. It is much harder once the construction is already made. Thus changes are encouraged during the initial prototype construction phase.

The prototype construction begins with the process of knowledge acquisition. The expert and KE sit together to guide the developer in learning about what are their expectations. The key concepts, different classes involved, relationship between those classes, processes involved and how those processes are controlled, all are determined during this phase. The basic problem solving approach is decided next. The KE does an intensive investigation of the domain and learn about important aspects. This process also involves looking for the information available from documentations, interview videos, logs, articles, training materials, case studies and so on. The KE also checks if it is feasible to have some machine learning approach to handle that part. The KE meets the expert once he think he has substantial amount of understanding about the domain. Meeting and discussing with expert helps him clarify if he is on the right track or not.

Here is a sample questions KE would prefer to get answers of.

1. What are the decisions that Expert generally make? For example when he decides that a packet needs deeper inspection.

2. What are the impacts or outcomes of those decisions? Once the expert decides a packet needs deeper inspection, how the packet is to be tested and what are possible ways the packet is further analysed and how decisions are made based on the results.

3. What are essential ingredients to take that decision? The ingredients include resources, particular conditions for a specific outcomes, or a set of events which can influence that decision. How expert decides that the packet needs deeper inspection. Does he look at the packet header? Does he look at the specific value? Does

1  Prototypes are also used for other conventional systems which are harder to visualize. he check for a typical combination of port number and IP address or a pattern? Does he continuously check for something over a sequence of packets or a typical sender or a typical pattern of attack?

4. What the past record indicates, how consistently the outcomes follow this decision making process? How many times the deeply inspected packet really result into catching an attack? What are the chances that the expert’s decision to deep inspect also misses out an actual attack? In other words, taking the stock of normal false negatives and false positives. This will help in assessment of the performance once the ES is developed w.r.t. expert’s own performance and also develop a matrix for ES performance assessment.

5. Are the evaluation experts and the KE are in sync for outcomes for given sample inputs? Do they agree on the fact that the packet under consideration really need a deep inspection?

The KE now decides about basic system ingredients, for example the KR scheme, the inference mechanism, and the design of forms for input and report formats. The form design is provided to normal user to test if he can read and respond back properly or face any issues. Reports indicate how the normal user interprets the response from ES. Ideally, the form should mimic expert’s consultation model. Thus it should include everything an expert would like to know when invited to solve the problem first. Once that part is decided, the KE looks for alternatives to implement that model. Current trend is to use the same general purpose language which is used to develop other parts of the system. It is imperative that the KE must put all his understanding in this model and allow the expert to critically examine it. Even if KE’s understanding is wrong, it is caught and improved in this phase. The prototype implementation is completely tested here.

Some of the most common problems of the domain are addressed by prototype. When KE and expert work in sync to validate the prototype, the results indicate the choices made during prototype being right or wrong. Issues like whether the knowledge representation is right, the rules written represent the idea of expert correctly, whether the implementation is able to handle the cases presented to it in real time and so on are also tested.

The problems addressed by prototype is usually the most common of the domain. The KE and expert has to work in sync to validate the prototype. Again the prototype is analysed on technical grounds to assess the methods for KR, implementation and other related issues. One of the major outcomes of the prototype construction is the modification of the problem statement. Two more usual outcomes are KE have at least entry level knowledge of the domain while the expert has more clear idea about what is lying ahead and what is his role in the process.

The knowledge engineer identify the sources from where the knowledge is to be gained, for example reference books, names of RFCs, the methods for calculating various things for example fragmentation and retransmission time etc.

Often the experts do not know exactly how they solve problems and this phase gives them their first experience of verbalizing their knowledge. Normally KE and expert both together take up various case studies to learn about typical solution sequences and then decide parameters for prototype generation and extension. For example when the user complains about slow network connection, the expert might check the connection, run Wireshark to test if TCP connections are timing out, is the OS run spurious services, are the software patched and is of latest version and so on. In medical domain, when the patient complaints about fatigue, the expert checks for BP, blood sugar level, age, patient’s and family history and few other things to start with.

Sometimes the problem solution is diverted to wrong directions. For example while developing an ES to catch an insider attack, it is quite possible that the designer starts working on enumerating all possible attacks which may be useful but not all that important. The user might waste lot of time and energy unnecessarily. The conceptualization helps all stakeholders that this should not happen. For example in medical domain, when a patient is in comma, some tests indicate brain haemorrhage, there is no point in checking the heart or lung related issues. If the developer is designing a system where it does something as weird as this, there is something wrong. The prototype construction phase helps both KE and expert realize if this is happening and avoid it.

Though we have seen that the ES development is usually done using conventional tools, specific decisions like using rule based ES or neural network based or having a neuro fuzzy component are still to be made. The conceptualization process throws light on that part as well.

Formalization

Once the decisions about the basic ingredients is made, the system development part starts. The first step in development is the formalization. This step begins with recording and organizing the knowledge gained during prototype development. Sometimes one needs to group rank and order knowledge, finding out missing points, remove redundant or unnecessary part and so on. At least a tentative plan is made before the actual implementation begins. Implementation steps for the system also is decided and recorded. Tentative deadlines for different milestones are set and recorded too. So far expert and KE are involved in the process, now other stakeholders are invited and informed about the system. The idea is to provide more visibility for the current understanding of ES. One of the important outcome is also to provide various checkpoints and milestones and their deadlines. Finding out which activities can happen in parallel, allocating those tasks to human resources and analysing the complete project as a whole are all involved in this process. The complete understanding gained during earlier steps is now formally represented for developers to start working at.

The first step in formalization is about analysing the problem in detail and generate a complete definition. Definition includes functional description of each module, objectives and problems to be solved (for example should the ES for intrusion detection be just detecting and reporting to admin or should it respond back to some extent, should the ES work only on the main servers or cover the entire network etc. One important ingredient is restrictions if any. (For example the detection process must not account for more than 5% of total traffic in a normal situation). A critical component is the expectations of the normal user (most of the times same as customer), as meeting the customer’s expectation is the primary goal for development of any system leave alone ES.

Once the detailed study is complete, the next part of formalization process include various plans for the design including project, test, product, support and then implementation planning. The design process completes with a design document containing a complete description of a tentative design.

The traditional design for conventional systems are very detailed to the extent that all functions defined with parameters and return values and complete module hierarchy, database design and complete description of all fields are readily available for the code once the design part gets over. It is not possible to do the same while working on ES. Domain knowledge is not well understood at this stage to finalize that right now. A more flexible and ‘let it evolve’ approach is really needed. However, the knowledge representation, inference and overall architectural design is developed during this phase. The detailed decomposition into finer details is usually done in the implementation stage.

One interesting problem might occur at this point of time. One may find the design document has major discrepancies as compared to the prototype design. In that case, a better way to reiterate the process, redesign the prototype and then reconsider the design process.

The formalization process also designs program logic using various methods for visual demonstration. Many a time conventional diagrams are used to represent the problem solving steps. Another point is to make sure that KR is designed keeping in mind credibility and acceptance by the normal user (or customer) . For example questions asked and rules examined should follow the same sequence as that of human expert. If there is a model behind the decision making process, it is to be discovered and modelled in this phase. For example in intrusion detection, human experts often use their psychological knowledge to determine what the attacker is up to and what is the meaning of his actions. Characteristics of the information to be used for gathering and problem solving is also ascertained in this phase. For example uncertainties are to be clearly understood by KE. The knowledgebase design takes place in this phase. This additionally demands knowledge chunks to be designed and their relationships to be identified. Also, it is also expected to provide a more accurate user interface than the earlier stage.

Now let us look at the planning process in brief.

Project planning

Like conventional systems, the project planning includes resource requirement analysis, budget in terms of time and money and a possible description of the milestones and deadlines as a schedule. Some diagrams indicating relationships between tasks and their criticality are also provided. Like conventional systems, various human resources capable to handle the given task are also assigned during this phase.Unlike the conventional systems though, the most important resource, the expert’s utilization is given the highest priority. As and when the expert is available, the project is scheduled to take his advantage.

Test Planning

As we have seen in the previous module, the testing part is handled by another team of experts. The design test cases to validate the system once developed. They also design procedures to apply those test cases. The idea is to provide test cases for normal cases as well as abnormal situations. The skills of the verifying expert is in to design complicated cases which the ES might fail to address.

Product release planning

Like other commercial systems, alpha and beta releases are made before final release. The alpha release is for the closed door company employees and beta is for others interested. After each phase, the revisions are provided to make sure that all concerns shown by users are properly addressed. Sometimes even multiple such rounds are done. Only after substantial level of confidence is achieved, the system is ready for final release.

In fact the product planning also involves product evaluation, which happens during the normal run. The system is continuously tested to be producing correct results, checking if there is any abnormality observed, also the extension ideas provided by users are considered. The primary idea is to assess the usefulness of the system and retain the confidence of the user. This phase involves clear planning for how these things are carried out once the product is released.

Support planning

After product planning, support planning is done. Once the system is out, the users start working, they start facing problems not expected or at least encountered before. Training the users to showcase the functionality and how to use that functionality are two common objectives of the support. This includes how users can communicate problems to the company and who will be responsible for handling them, what type of support staff is needed, how system configuration and implementation issues are managed, how domain experts are used to provide further support and so on. It is also important to assess side effects of this support activities. Usually test cases for all functionalities are readily available and are executed immediately after the changes made. If there is any deviation, it is addressed.

Implementation Planning

The final planning is about implementing the system. This process determines how the development process will take place. This will describe how the knowledge is extracted including deciding about meeting with experts, database and input forms design and other similar programming and implementation issues. Usually the ES is developed as a segmented and incremental model. The system is developed segment by segment, tested individually and then integrated with the rest of the system.

Implementation

In this round, the KR system is modified to reflect the design decisions. For example it is found that some classes thought of earlier to be part of the system are found not very useful or may be decided to be combined together. It is also the case that the class designed are to be elaborated and a new hierarchy is to be added and so on. Clearer understanding of the domain now might result into better field design for classes, restructuring of the database tables and fields, change in module hierarchy etc. One technique proved useful is to divide knowledge into manageable segments. That is known as knowledge partitioning. For large ES, it is more than essential. It is found that large knowledge segments are hard to manage and even harder to maintain at later runs. The knowledge chunks (which are logically near to each other) are grouped into segments of the knowledgebase which are more or less independent of each other and represent a logical entity. Connecting these knowledge chunks into segments is also known as partition base. Such design is usually made at the architecture level but they are equally important to be considered at the implementation level.

This state transfers the formalized knowledge into the code. Usually the first job is to have a working prototype. The knowledge chunks, rules and the methodology for applying rules (the control strategy) are all formalized and executable module is prepared. The role of KE in this process is pretty crucial. Choice of program development tool is a critical part of this phase. Everything that is modelled in earlier phase takes the physical shape now.

A well-known SE principle of cohesion and coupling applies here as well. Cohesion is extent to which different knowledge chunks are tied in a given segment logically. The idea is to keep closely related chunks together in a knowledge segment and thus increase the amount of cohesion. Coupling is the connections between segments which should be less if the segments are truly independent. The idea is to make sure the inter-segment connections are as less as possible and thus reduce the coupling.

Once this is done, the prototype is revisited and the development phase begins in the earnest. The basic framework is prepared first. The basic framework includes inference engine, the minimal user interface, etc. Once the basic framework is ready, it becomes easier to add new classes, rules, facts etc. to it. Now core knowledge base is to be constructed. Rules and facts are added. These knowledge chunks are decided based on their requirement. The requirements are decided by the test cases provided by verifying experts. Any ancillary part needed for proper execution and testing is also provided.

Researchers have recommended an iterative method for building knowledge base. It works like this

1. List of cases are designed first

2. Empty knowledgebase as well as empty case validity test list is initialized

3. For each case, the knowledge is extracted from the expert.

4. If the knowledge is repetition of something already present, ignore it

5. If the knowledge is either more general or little different than the one that is existing, the existing one is modified to accommodate. If needed, the hierarchy also is modified.

6. Testing the validity of that case is also designed based on expert’s input

7.  The case validity code is added to case validity test list

8. Now the new case is integrated with the existing knowledgebase

9. After the new case is entered, first case from validity list is picked up

a. Case validity test is run to see if that test still runs the same

b. If yes pick up next test; if list is over go to 10, otherwise go to a

c. If the test case fails, that knowledge part is reiterated and it starts all over again.

10. The knowledgebase is now ready for use.

A diligent reader might pose a question, why ALL cases are tested when a new case is introduced? It is because when we add new rule, or fact, it might have undesirable side effect on any other case. It is hard to know which case is effected by which change. Ideal solution is to test the knowledgebase immediately after inserting a new case. Whenever it is found that the knowledgebase has become inconsistent, it is modified and all test cases are fired once again.

Once the knowledgebase is ready, it is connected with the database and front end. Those routines, which are not part of mainstream development project, are developed in parallel with the knowledgebase and now fused together. Cloud integration, remote server connections, client server deployment decisions are all taken at this point of time. Sometimes even number of tiers needed are also decided based on system requirements.

In this phase, the formalized knowledge is mapped into the code using the language or framework chosen. KE and development team have to work hard to build a working prototype. The complete knowledge structure including class hierarchies and programming modules with inference rules and control strategies are built. It is also sometimes necessary to look for things which are missed out are identified and handled.

The final thing is to integrate everything into a single comprehensive system. All interfaces are tested, tests which involves complete system are executed. Both KE and expert must measure the success rate of the system and closely assess why system is not able to solve problems if it is so for some cases. Once they are satisfied, the verifiers will start working on the system and verifies it with their own case studies. That happens in the next phase.

Testing and Evaluation

Once the implementation is over, the evaluator starts complete evaluation of the system. One interesting difference between a conventional system and an ES development is that testing cannot be done on on/of scale for ES. There may be more than one correct responses and none of the responses are clear winners over other. Also there are more than one ways to solve problems with a few cases favour one and others favour other aspects of the solution. The Mycin case was discussed before. Evaluation is never considered in absolute terms. A performance of ES is always considered against the performance of well-known experts of the field. In fact neither experts nor ES found to be having 100% efficiency. So called ‘second opinion’ is also a good idea when one uses ES!

The ES is also generally evaluated on user friendliness, completeness, database consistency, incorrect inputs and so on like other conventional systems.

During the testing and evaluation the designer also drafts a long term maintenance and extension plan. The idea is to have user’s inputs and provide more general functionalities, provide corrections to knowledgebase if needed, extensions or modifications in class design or database design, adding missing knowledge chunks, sometimes expanding the domain itself or even changing some fundamental part as other experts are invited or the primary expert himself is feeling for fundamental changes. The last statement might surprise some but building an ES is also a learning chance for the expert to learn about his knowledge and skills to articulate it. After working with the ES he might realize that some fundamental part requires modification for better results and he might ask for it.

One of the important outcome of this phase is a design of the long term maintenance plan. Modification of knowledge must be anticipated and the implementation design must the able to accommodate further changes. Proper documentation is equally important like conventional systems. The interface to the ES for the customer is to be designed in a way that explanations for the users is readily available. Extensive plan to help the user for using the ES is a must.

One more consideration is newer versions of the software tools used to develop the ES. When the newer version of software available, many times a simpler and faster way to do something is available. Changes which takes the benefit of the facilities provided in latest versions is an important consideration.

Apart from user interface, this phase also should deal with other tiers like database and business logic layers and operating systems sometimes. That is also planned in this phase. An important aspect of testing phase is performance assessment which is discussed separately in the next section.

Performance assessment

Though testing is the last phase, it is more crucial than most other phases. It is not only about finding and solving syntactical or normal logical errors but many other things. First, it must check the performance of the program with clearly designed test cases. In some cases there are datasets which can help us test our program (many medical domain have huge datasets on which researchers can test their program, KDDcup is another example which is used in Intrusion Detection process). The program’s performance w r t manual process is also a must. Verification of all normal cases being handled and also with border cases is the minimum requirement of this phase. It should also test for contingencies. For example an IMD (Implantable Medical Device) programing requires to handle a case of emergency when patient is unconscious or not in a position to communicate. The test cases must include all possible concepts that the ES is designed to address.

Unlike conventional software, the ES must manage uncertainties. Carefully designed test cases can test if the ES is capable to handle uncertainties in rational manner. For example a driverless car must manage to take right decisions at the time of accident or when there is a choice to be made and none of the option is good. For example when the brakes stop working, the car can decide moving further on Highway or collide with a tree to halt. If the car chooses the second option, it is not a good decision but better than continue running and hit innocent people.

There may be a few iterations but a solid set of test cases might help finding out all possible lacunas and help the program consistently provide responses at par with human experts. If the designer takes serious pain during this process, there is a far better chance of the system to be accepted by its customers.

The performance assessment process should test for correct execution of all rules, check if no case remain unattended, correct answers to all test cases are provided, the system act consistently and there is no case which surprizes both expert as well as the customer. That means the ES must be tested for completeness, correctness and consistency. The testing phase must also check if the control strategy chooses the right rule to apply in all cases, information needed for decision making is available at that time (for example the medical diagnosis system must check if the patient has heart ailment and require stent to be implanted, whether the patient has diabetes is verified or not2). Testing for all conclusions being logical and consistent with expert’s own judgement for same case is also a must.

Another important part of performance assessment is about sequence of question generation. If system generates questions (based on answers to previous questions), exactly like human experts to quickly drive the patient reveal information for the decision making, it is considered to have right performance.

It is also imperative that the system’s recommendations are also to be reconciled with human expert’s recommendation for similar cases.

Summary

We have looked last four phases of expert system development process in this module. We started with prototype construction. Prototype is important because of the abstract nature of the problem. Prototype reflects experts’ and KE’s understanding about the system. Once prototype construction process is complete, formalization begins. Prototype construction process has conceptualized the solution which is formalized in this process. Knowledge structures are now designed and knowledge is inserted in those structures. Formalization also invites planning for the rest of the process including implementation and maintenance. Unlike conventional systems formalization process does not detail the process to finer levels. That part is left to implementation phase. A critical component of the formalization process is dealing with uncertainties. The next phase is implementation which involves coding the expert system. Class structure, database design etc might be changed or modified, knowledge chunks are grouped into segments, and iterative class based implementation with case validity for each case is deployed. Once the knowledgebase is ready, it is connected with front and back ends. Once the working prototype is ready, the testing and evaluation process begins. As multiple correct outputs are possible, it is hard for evaluator. Performance assessment is one of the important aspects of the testing process.