Executable Specifications: Approach, Tools and Benefits

Ideally, requirements would be written in a way that makes it easy for the development team to verify the system against them. Making them executable enables the team members to run them as often as necessary. Executable specifications can be based on stake-holder defined stories and are therefore a means for improving communication and clarifying interpretation.

The benefits to this approach are very compelling. Firstly, there is a much tighter link between the customer and the development team. Secondly, it provides a tool that helps reduce the gap between what your customer wants the system to do, and what the system really does.

RSpec is a Behaviour Driven Development framework for Ruby providing a story framework for describing application behaviour and a spec framework for defining object behaviour.

FitNesse is another tool that is based around a Wiki that enables the collaborative definition and execution of tests.

Both tools present the tests in a form that is more user-readable than your average unit test. This allows stake-holders to see and understand the tests as well as seeing and understanding the test results. In theory it also allows stake-holders to write tests, but in practice this rarely happens.

Engineering Design Verification

Numerous computational tools exist for verifying the structural integrity of an engineering design. These tools are generally based on either finite-element or classical analysis methods. While these tools are very effective in verifying design conformance against specific engineering requirements, they do not provide an effective means of either representing or verifying the design against stake-holder defined requirements.

Additionally, there are tools emerging that support automatic verification of 2D drawing and 3D model representations of a design. An example is the iCHECK product developed by INCAT. iCHECK can help organizations minimize the risks associated with poorly-produced CAD data by applying a series of automated checks.

The available checks include:

• Model meta-data (filename, properties, tags, associations, …)
• Modeling practices (properly constrained sketches, allowed features, feature order, sizes…)
• Model annotations (dimension styles and overrides, line types, colors, …)
• Model visibility setting (render settings, part visibility, …)

One of the benefits of a tool like iCHECK is that a stake-holder can define their own checks and share this directly with the engineering design team enabling them to automatically verify the design for themselves. Although this approach can be effective, it does not verify the design against the actual stake-holder requirements for the part or assembly. This is because the tests focus on verifying the model representing the design rather than the design itself.

Many CAD systems have attempted to integrate tools that capture engineering product knowledge and design rules directly into the model. An example is the CATIA Knowledgeware suite of tools. These tools have focused on the capture and application of business knowledge in an attempt to automate or generatively create, CAD based design data. Unfortunately, the business knowledge required to make these tools effective is generally not available in a suitable form. Additionally, these tools have lacked the technical maturity to be used reliably and the captured knowledge is not in a form that can be persisted or repurposed separately from the tool.

There are two problems that these tools are trying to solve:

1. Automated/generative design based on product knowledge; and
2. Engineering design verification based on product knowledge.

For the solutions to these problems to be effective, there needs to be a recognition of the importance of stake-holder involvement in the capture and execution of the business knowledge (or requirements) that drive the engineering solution. It is often the case that as an organization attempts to capture engineering knowledge a number of recurring problems emerge. Some of these include:

1. Staff generally resist documenting knowledge as part of normal working practices and are even less enthusiastic in supporting specific knowledge led initiatives.
2. Staff tasked with eliciting and documenting the engineering knowledge are too inexperienced to appreciate the significance and implied context.
3. Lack of a commonly used and recognized standard for storing and persisting knowledge.
4. Lack of commonly available tools for automating the capture and persistence of engineering design knowledge.

Note: The last two points may be challenged by those with specialist skills or experience in knowledge management – but I would argue that the lack of industry acceptance suggests that there are still significant opportunities for improvement.

So how could these tools be improved? Could the Behaviour Driven Development approach used in the software industry be applied?

Acceptance Tests as Engineering Design Requirements

As introduced earlier, acceptance testing frameworks are gaining momentum and provide a medium for stake-holders and software developers to communicate a common understanding a how a system should behave. Therefore, one approach would be to facilitate the definition of stake-holder requirements using a structured software approach. This would enable the definition of acceptance criteria for various aspects including structural integrity (strength, stiffness, service life, …), geometry (interfaces, fits, space-envelope, …) etc to be captured. The output from this could be a series executable tests that could be shared by the stake-holders and the engineering designers to run automatically to verify the design as it progresses.

So what would these test look like?
Basing the syntax on RSpec, the tests would be defined as plain text stories.

Story: prevent vibration induced driver fatigue
  As a vehicle driver
  I want to reduce vehicle vibration
  So that I get a quite and comfortable ride

  Scenario: vehicle driving on dirt road
    Given my vehicle has Velocity of 60km/hr
    When my vehicle has RoadRoughness of 1000mm/km
    Then the Time to reach a VibrationDoseValue of 8.5m/s1.75 is greater than 6hrs

  Scenario: vehicle driving on sealed road
    Given my vehicle has Velocity of 60km/hr
    When my vehicle has RoadRoughness of 200mm/km
    Then the Time to reach a VibrationDoseValue of 8.5m/s1.75 is greater than 12hrs

As you can see keywords like Given, When and Then would then be translated into software based tests. Additional domain specific keywords (Time, Velocity, RoadRoughness and VibrationDoseValue) together with the associated units (hrs, km/hr, mm/km and m/s1.75) would also be used in the generation of test cases.

There are however a number of enabling technologies needed to make the suggested approach possible. These include:

• A domain specific language or standard for defining engineering acceptance tests similar to that demonstrated above
• A testing framework enabling the execution of tests.
• Closer integration and interoperability between CAD, FEA, CFD and other modeling or analysis tools to facilitate test execution.

As far as I am aware there are currently no initiatives that address items 1 and 2. Progress towards the last item is largely a commercial issue with different engineering software vendors unwilling to trade competitive advantage for data interoperability. Standards for representing engineering data such as ISO 15926 and ISO 10303 are of benefit, however vendor adoption is slow, especially for more advanced definition of geometric features.

Finally, the relationship between engineering industry standards like Vibration Exposure – could be used to form the basis for a series of executable specifications. Together with the normal textual definition of compliance criteria, a set of software based test codes could also be provided to assist stake-holders and engineering designer to verify designs.

Small Regular Deliverables

At the heart of agile is the incremental delivery of projects in small regular intervals, rather than one large delivery at the end. Each of the delivery iterations is a subset of the entire development lifecycle and is demonstrated to the customer. This has the effect of reducing the risks associated with the development progressing in a direction that is not consistent with customer requirements. Applying this principle to engineering design does have some limitations, particularly since the cost and complexity of construction usually prohibits multiple iterations. However, within the design phase of an engineering project, there are good opportunities to use this approach and it is this part of the product life cycle that has been considered.

During the design phase of an engineering project, milestones are often introduced to mark preliminary and critical phases of the design. These milestones do encourage incremental delivery, but they are neither small enough, nor regular enough to be consistent with agile principles. Modern CAD tools enable rapid development of digital mock-ups that can fulfil this objective. The CAD model can contain sufficient design detail to accurately represent the product and surrounding structure. As a result the CAD model can be used as the basis for verifying many different acceptance criteria. Additionally, rapid prototyping systems enable the creation of physical mock-ups that greatly enhance customer engagement and interaction with the design process.

From an agile perspective these deliverables are important for customer engagement and verification of the design direction. However, prototypes that do not add directly to the final product are not valued as highly as actual progress toward the final delivered product. That said, there is still significant opportunity to make incremental deliverables part of the design process and no technical reason why this cannot occur throughout this phase of an engineering project.

Deliverables Measure Progress

Customers often propose an earned value schedule for a project that is based on deliverables and is linked to either financial payments, or the release of additional project funds. These types of earned value measures are compatible with agile principles, particularly if they are granular enough to encourage close customer involvement in the project.

For engineering design projects, the CAD model is increasingly becoming a digital representation of the complete product including geometry, materials, manufacturing details and even the original design intent. Measuring the progress of CAD model maturity and conformance with acceptance criteria will provide the most agile measure of project progress.

Customer Collaboration

Customer satisfaction is considered important in engineering design projects, but the value of customer involvement in the development and review of the design as it progresses, is not widely utilized. Many project resource models fail to consider the benefits that can be achieved through close engagement with the customer, even for tasks that may appear well defined and understood.

Traditionally, the approach has been to develop highly prescriptive specifications and then negotiate changes with the customer using forms and documents such as Engineering Change Notes. In an agile environment, changes are driven by the customer and embraced by the development team. During the design phase of an engineering project there appears no reason why the same approach cannot be used. Additionally, when collaboration occurs between the customer and the engineering design team, the customer will tend to accept a greater level of responsibility for decisions as they can clearly see their impact on scope, schedule and cost.

The Scrum model advocates meetings that occur daily and at the end of each delivery iteration or sprint. The daily meetings involve the customer and all team members and are an opportunity to discuss each team member’s progress, plans and any obstacles. The iteration review and planning meetings also involve the customer and is their opportunity to prioritise the order in which features are implemented. This ensures that the development team are always working on the features that are of the highest value to the customer. These meetings may not always be practical because of team size, geographic locations or prior commitments; however, even at a small scale they have significant value. In large projects where the majority of engineering design work occurs in a separate location, a daily telephone or internet conference call with all team members will achieve a similar result and is a technique that is very valuable.

Simplicity and Design

Starting with the simplest possible solution that satisfies customer requirements and then maturing the solution during course of the project, is a common agile practice. This approach is somewhat at odds with software architecture design principles, which often need to be addressed early in a project. The conflict is typically resolved by developing software in such a way that it can be re-structured as easily as possible.

The main techniques for achieving this include: maintaining loose coupling between components and implementing automated tests. The loose coupling provides the opportunity to easily replace components or restructure the overall design with minimum impact. Automated tests allow architectural or component changes to be verified quickly thus further lowering the cost associated with changing and evolving the solution through the course of the project. This process is often referred to as refactoring.

For engineering design projects, implementing the simplest possible solution is an implicit assumption for most engineers and therefore compatible with agile principles. What is lacking however, is automated testing that verifies design changes. Significant work has already been done to automate structural analysis methods using both computational and classical analysis methods. However, this analysis is generally performed manually as a supplementary activity, rather than a concurrent activity that is automated. Therefore an opportunity exists to implement automatic design verification against customer defined acceptance criteria.

Embracing Changing Requirements

Traditionally, engineering project management practices have emphasized the need to control change. This is partly because project change has been closely associated with project cost, especially within a fixed price environment. Conversely, agile principles are based around an assumption that change will inevitably occur during the course of a project and that it is better to position the solution and working practices so that they can easily adapt. Furthermore, in an agile project it will be necessary to compromise at least one of either project; scope, cost or schedule and customers need to appreciate that they are responsible for guiding this compromise.

The agile practices of refactoring and automated testing help, software projects adapt to change, while the close customer involvement ensures that changes are identified as early as possible. For engineering design projects these principles can be applied as equivalent technical solutions exist, however the costs associated with automated testing can be very high. Consequently, engineers typically question the return on investment associated with automated verification. Additionally, engineers often take a conservative approach to alternate design methods and therefore changes are typically met with strong resistance. That said, there is generally a trend towards engineering automation and this is likely to evolve into automated testing.

Pair Programming/Working

Pair programming has attracted controversy because managers typically assume that having two developers implement the same functionality is only half as efficient as conventional working models. As an agile practice, pair programming has the effect of increasing software quality through peer review. Quality is critical as it ensures that the resultant software is in a state where it can be adapted to meet changing or additional requirements.

In a typical engineering design environment, the practice of pair working is uncommon. This maybe in part because of how tasks are broken down and matched to people with appropriate skills and experience. Another reason maybe the perceived cost ineffectiveness. Either way, the clear benefits that pair working provides should justify its adoption more widely.

With this in mind, there may be better ways to decompose engineering design tasks so as to exploit the skill and experience combinations that pair working offers. An example could be pairing a structural designer and analyst, instead of having the designer initiate structural layouts and the analyst size the structural features. This would encourage a more concurrent approach and result in a higher quality design and less rework.

Summary

Agile methods have become the default approach for software development projects. Many of the agile practices and principles can be more generally applied to other disciplines including engineering design. The opportunity for engineering design organizations is to improve the reliability of project success and the quality of resultant designs.