Difference Between Design and Operational Maintainability Questions Read the questions! Some have multiple parts. Think about the questions and research as

Difference Between Design and Operational Maintainability Questions Read the questions! Some have multiple parts. Think about the questions and research as necessary. Your answers should be at least a 1/2 page for each of the 2-point questions and one page for each of the 3 and 4-point questions.

PART I: Test & Evaluation Basics

1. What is the difference between “Test”, “Evaluation”, MOE’s and MOP’s? What is the relationship between MOE’s and MOP’s and what role do they play in the T&E process? (2 points)

2. How does M&S fit into test and evaluation? What needs to be considered when using M&S in DT & OT? (2 points)

3. Define the following terms and give an example for each:

a. Operational Suitability

b. Operational Effectiveness

c. Lethality

d. Survivability

If possible, use a commercial example for each. (3 points)

4. Test points are critical to a successful test program. What considerations need to be taken in account when choosing test points? (2 points)

5. What is the purpose of test metrics? Give three examples of test metrics and a brief description why you feel these are important items to track. (4 points)

6. Net-Centric systems pose challenges to systems integration and test. Why? Provide 4 examples of challenges with a brief explanation for each (4 points)

7. Briefly, describe how logistics testing gets integrated into the system life cycle and why? (4 points)

8. Briefly describe the relationship between reliability, maintainability, and availability. What is the difference between Design Maintainability and Operational Maintainability? (4 points) Lesson 7
Systems V&V Methods: NonTesting
VVT Activities and Methods
Reference: Engel (2010). Verification, Validation
and Testing of Engineered Systems, p. 34.
SYS 511 – Ver 2.1
2
Discussion
 Non-testing methods include:
 Activities that are not actual testing – such as planning and participation in
reviews
 System Test Simulation
 Design of Experiments
 Software Inspections
 We have discussed many non-testing methods already and will be
examining System Test Simulation, Design of Experiments and
Software Inspections
 Areas we discussed in class or were discussed in SYS501 include:
 Requirements Verification Matrix
 Requirements testability
 Planning & Scheduling
SYS 511 – Ver 2.1
3
Lesson 7a (part 1)
System Test Simulation
4
SYS 511 – Ver 2.1
Some Definitions
 Model
 A mathematical representation of an object (a part, a product, a machine,
a facility, an organization, etc.) or a process (e.g. a specific manufacturing
or business process)
 A mathematical model characterizes the behavior of its subject through the
form of the equations chosen, the variables and parameters present, and the
ranges or values of those terms for which the model is considered valid
 Simulation
 Simulations are the specific application of models to arrive at some
outcome
SYS 511 – Ver 2.1
5
Simulation Classification
 Depending on the type of data available we classify simulations based on
how they will be built.
 Top Down – simulation is constructed from mathematical models that are know
to capture the system’s behavior.
 System behavior obeys an unsolvable mathematical model
 Use numerical methods to approximate original equations
 Used for complex physical systems
 Bottom Up – build a virtual system reflecting real behavior as much as possible.
 System behavior is known statistically or empirically
 Governed by dynamic inputs and rule based or probabilistic principles
 Used for a system of production and distribution or information flow in an organization
 Indirect – concentrate on simpler models that capture the characteristics of a
system.
 Used where system behavior is not fully understood or too complex to model directly
SYS 511 – Ver 2.1
6
Simulation Classification
 Can also classify based on how they are constructed.
 Dynamic vs. Static – Dynamic includes the passage of time and
examines state changes as they occur. Time plays no role in a
static simulation.
 Continuous vs. Discrete – In continuous simulations the state can
change continuously over time. Discrete simulations change only a
points in time.
 Deterministic vs. Stochastic – Deterministic simulations have no
random input. Stochastic simulations operate with at least some
random inputs.
SYS 511 – Ver 2.1
7
Test Simulation Classification
Discrete
Static
Continuous
Dynamic
Deterministic
Stochastic
Engel – Table 4.23
SYS 511 – Ver 2.1
8
More Definitions
 Simulator
 Machine for simulating environmental and other conditions for purposes
of training or experimentation
 Stimulator
 Stimulus that incites to action or exertion or quickens action
 Emulator
 Computer system that duplicates the functions of one system using a
different system, so that the second system behaves like (and appears to
be) the first system. The focus is on the exact reproduction of external
behavior.
SYS 511 – Ver 2.1
9
Simulation Types
 Constructive
 Constructive simulations are strictly mathematical representations of
systems and do not employ any actual hardware.
 They may incorporate some of the actual software that might be used in a
system.
 Early in a system’s life cycle, constructive simulations can be expected to
provide the most system evaluation information.
 Virtual
 A simulation involving real people operating simulated systems.
 Inject man-in-the-loop in a central role by exercising motor control skills,
decision skills, or communication skills.
 Live
 Training exercises and other live simulations provide a testing ground with
real data on actual hardware, software, and human performance when
subjected to stressful conditions.
 These data can be used to validate the models and simulations used in an
acquisition program.
SYS 511 – Ver 2.1
10
Where does it fit?
SYS 511 – Ver 2.1
11
Simulation Spectrum
SYS 511 – Ver 2.1
12
Model-Test-Fix-Model
 A process that integrates M&S into the systems engineering processes
for the purposes of iteratively evaluating and improving both the
system‘s design and its models.
 When live testing data is collected, it should be used to help
recalibrate the M&S to ensure its trustworthiness for future use.
 To collect the appropriate data to help validate M&S, one must plan
ahead and identify the data to be collected before live testing begins.
SYS 511 – Ver 2.1
13
Model-Test-Fix-Model
SYS 511 – Ver 2.1
14
Not the perfect answer
 Not a total substitute for live testing
 While we are getting better, not everything can be simulated
 Can be used to complement and validate live T&E
 But…complex simulations can be more effective analysis tools than
limited live testing.
 Missiles
 Nuclear testing
 Environmental models
 Ocean environments
 Atmospherics
SYS 511 – Ver 2.1
15
When to use M&S
SYS 511 – Ver 2.1
16
Uses of M&S
 Operational:




Planning
Training
Operations Management
Decision making
 Quantitative:





Measurement
Monitoring
Prediction
Evaluating
Tuning
SYS 511 – Ver 2.1
 Analytical:




Experimentation
Troubleshooting
Process Design
Process Improvement
 Qualitative:





Illustration
Explanation
Understanding
Discussion
Consensus Building
17
Why Use MS&A?
 MS&A can provide a tool to address complexity
 Provide structure to decision processes
 Improve understanding of processes to be supported
 Shorten design time and allow for assessment of multiple design options
and effects of change
 Provide a venue to explore effects of specific ‘events’ on processes and
systems behavior
 Allow for early and continuous assessment of technical feasibility
 Improved articulation of plans to solicit input from multiple stakeholders
SYS 511 – Ver 2.1
18
Advantages & Disadvantages
Advantages
Disadvantages
 Can shorten schedules
 Return on investment
 Can gain deeper knowledge of system
behavior
 Results misinterpreted
 Flexibility in evaluating alternative
solutions
 Capturing subtleties of reality – a
subset of reality
 Repeatability
 Improves processes and products
 Exploits past experience through
model reuse
SYS 511 – Ver 2.1
 Difficulty validating model
 Discipline to stop
 People and organizational
commitment due to complexity
19
Verification, Validation & Accreditation
 Verification
 The process of determining that a model implementation accurately
represents the developer’s conceptual description and specifications.
 Validation is the process of determining:
 The manner and degree to which a model is an accurate representation of
the real world from the perspective of the intended uses of the model.
 The confidence that should be placed on this assessment.
 Accreditation
 The official certification that a model or simulation is acceptable for use
for a specific purpose.
SYS 511 – Ver 2.1
20
Validation Tools
 Benchmarking: Comparison of simulation outputs with outputs of
another simulation that is accepted as a “standard”
 Face Validation: Comparison of simulation design and outputs (under
well defined conditions) with the expectations and opinions of subject
matter experts in the simulation of interest
 Results Validation: Comparison of simulation outputs with the results
of test measurements made under identical input conditions
 Sensitivity Analysis: Determination of the variation in simulations
outputs for measures changes in inputs, functional operations, or
other conditions (generally used to supplement other validation
methods)
SYS 511 – Ver 2.1
21
Validation Tools – Assessment
 Benchmarking: Excellent tool if “standard” is available
 Face Validation: A “must do” but may be subjective
 Results Validation: Sufficient results from properly instrumented and
representative test data often lacking
 Sensitivity Analyses: May demonstrate that a particular input to the
simulation has very little impact on the outcome
-ORIf the input does matter, one can appropriately articulate the
limitations of the simulation and caveat the results
SYS 511 – Ver 2.1
22
M&S Planning
 Some M&S is complex and can take time to develop and validate
 Build into schedule
 Build into budget
 Goals and objectives of M&S in a program needs to be laid out early
SYS 511 – Ver 2.1
23
M&S Plan Table of Contents Example
SYS 511 – Ver 2.1
24
M&S Application to T&E (DoD example)
SYS 511 – Ver 2.1
25
Modeling and Simulation:
Enterprise ‘System of Systems’
 Traditionally M&S focused on individual components or systems
 A ‘system of systems’ in the enterprise can be modeled for many of
the same reasons we modeled individual systems
 Support design and design tradeoffs
 Assess performance
 Identify bottlenecks or interoperability issues
 Assess options for configurations
SYS 511 – Ver 2.1
26
Static Mockup
 A three-dimensional (3-D) model of a product or system that has no
moving parts
 Range in scope from an individual control, such as an airplane control
yoke, to an entire vehicle control station
 Types
 Foam core – Foam core is a thin sheet of dense Styrofoam TM (usually 1/8
or 3/16 inch thick), covered with white paper
 Wood
 Plastic – stereo lithography (aka 3-D printing) uses 3-D solid CAD data to
build parts from a liquid photopolymer resin that solidifies when exposed
to a high-radiance light source
SYS 511 – Ver 2.1
27
Dynamic Mockup
 A 3-D model of a product or system that has moving parts which do
not have functionality
 Range in scope from an individual control, such as an airplane control
yoke, to an entire vehicle control station
 Types
 Wood
 Plastic – stereo lithography uses 3-D solid CAD data to build parts from a
liquid photopolymer resin that solidifies when exposed to a high-radiance
light source
 Hardware to emulate moving parts such as controls and seat restraints
SYS 511 – Ver 2.1
28
Partial Task Simulators
 Simulating a specific aspect of the task for training
SYS 511 – Ver 2.1
29
Full Mission Simulators
 All tasks from start to end of a mission
 Recreates sounds, motion, visual scenes, instrument presentations,
and all other systems
 Full mission flight simulators
 Landing, takeoff, weapons delivery, night flight, formation flight and
cockpit familiarization in normal, adverse, and emergency situations
 Handling characteristics represent actual aircraft based on available flight
data and input from experienced pilots
 Full mission surgical simulators
 Pre-op, anesthesia, invasive activity, intervention, closure, and post-op in
normal, adverse, and emergency situations
 Patient responses represent actual human response based on available
medical data and input from experienced surgeons
SYS 511 – Ver 2.1
30
Lesson 7a (part 2) System Test Simulation
31
SYS 511 – Ver 2.1
Prototyping
 Prototype – A preliminary type, form, or implementation of a system
that serves as a model for later stages or for the final, complete
version of the system
 Generally experimental in nature
 Often with minimal documentation
 Usually incomplete
 Done more cheaply
 Done more quickly
 Prototyping – A hardware and/or software development technique in
which a preliminary version of all of the hardware or software is
developed and implemented as required
 Integral part of interactive, user-centered design
SYS 511 – Ver 2.1
32
Types of Prototypes
 Rapid Prototyping
 Quick development
 “Throw-away” mentality
SYS 511 – Ver 2.1
33
Benefits of Rapid Prototyping
 Visualization, verification, iteration, and optimization for design
engineers
 Communication tool for simultaneous engineering
 Form and fit tests of components
 Test samples for marketing studies of consumer design preferences
 Help in production planning
 Complement bid packages for tooling quotations
 Metal prototypes fabricated from polymer parts
 Tooling fabricated from polymer parts
SYS 511 – Ver 2.1
34
Rapid Prototyping
 In the “Software” world – quick turnaround development tools and
environments.
 In the “Physical” world there are currently five basic commercial
technologies. This include but are not limited to:
 Stereo Lithography (SL)
 Laminated Object Manufacturing (LOM)
 Selective Laser Sintering (SLS)
 Fused Deposition Modeling
 3D Printing
 Rapid prototyping is now used in the medical community to “dry run”
critical surgeries on patient models.
SYS 511 – Ver 2.1
35
3D Printing
 Term used to describe additive manufacturing techniques
 7 Types







SYS 511 – Ver 2.1
Vat Photopolymerisation
Material Jetting
Binder Jetting
Material Extrusion
Powder Bed Fusion
Sheet Lamination
Direct Energy Deposition –
36
Highlights of Stereo Lithography
 Called the “first” RP technique.
 Developed in 1986 by 3D
Systems of Valencia CA.
 A “3D graphics printer”
 Computer-Aided-Design 3D
model created.
 “Printed” on a stereo
lithographic printer
 Light sensitive liquid polymer
(resin) hardened by a laser
beam.
 Hardened in incremental layers
(layer increment defines the
granularity of the 3rd axis).
 Vat Photopolymerisation
process
SYS 511 – Ver 2.1
37
Stereo Lithography Example
SYS 511 – Ver 2.1
38
Laminated Object Manufacturing
 Patented 1988
 Sheet Lamination process
 Cross-sectional cutouts
fused together
 Paper, metal or polymer
sheets
SYS 511 – Ver 2.1
39
Selective Laser Sintering (SLS)
 Patented 1989
 Powder Bed Fusion process
 Fusing polymeric powders
with CO2 laser
 Accuracy 160 microns
 Polycarbonate, nylon, wax,
glass-filled nylon, powder
coated metals or ceramics
SYS 511 – Ver 2.1
40
Laser Sintering
SYS 511 – Ver 2.1
41
Fused Deposition Modeling (FDM)
 Patented 1992
 Material Extrusion process
 Robotically guided fiber
extrusion
 Accuracy 375 microns
 Casting and machinable
waxes, polyolefin, ABS
SYS 511 – Ver 2.1
42
Fused Deposition Modeling
SYS 511 – Ver 2.1
43
Material Jetting
 Material is applied in droplets through a small diameter nozzle, similar
to the way a common inkjet paper printer works, but it is applied
layer-by-layer to a build platform making a 3D object and then
hardened by UV light
SYS 511 – Ver 2.1
44
Material Jetting
SYS 511 – Ver 2.1
45
Reusable Prototyping
 Software
 Reusable code
 Evolutionary code
 Hardware
 Programmed Fabrication (Numeric Controlled Programming) develop once and reuse.
 Design may be refined with each use.
 Final product built from evolved program.
 Final product closely matches prototype.
SYS 511 – Ver 2.1
46
Types of Prototypes (Cont’d)
 Modular Prototyping (Incremental Prototyping)
 Adding new parts (prototypes) to existing ones to prototype a
larger and larger portion of the system.
 May link “Physical” prototypes with simulation.
 May play a part in system Integration as prototypes “stand-in” for final
component or sub-system.
 Horizontal vs. Vertical Prototyping
 Horizontal – Broad spectrum of product’s features but without
extensive functionality.
 Used heavily in interface design.
 Vertical – Exact functionality of a small part of the product.
 Helpful to validate functionality.
 Low-Fidelity vs. High-Fidelity
SYS 511 – Ver 2.1
47
Fidelity vs. Resolution
The degree of fidelity is the degree of emulation.
e.g. materials properties, embedded subsystems, signature characteristics, etc.
Greater fidelity = greater emulation = greater simulation cost
SYS 511 – Ver 2.1
48
Resolution vs. Fidelity
Greater resolution requires more computing power, and
communications bandwidth capacity if networking
Fidelity, Resolution, and the degree to which the model is true to the
Physics or Chemistry (aerodynamics, hydrodynamics, thermodynamics,
optics, acoustics, electromagnetics, etc.) consume computing power.
The more of each, the greater the M&S complexity & cost
SYS 511 – Ver 2.1
49
Resolution vs. Fidelity
Higher Resolution
SYS 511 – Ver 2.1
Which
is
better?
Higher Fidelity
50
The Spectrum from Lo-Fi to Hi-Fi
Low-Fidelity
Paper-based sketches
Storyboards
Computer-aided-design “layouts”
Slide Show or Video models / simulations
Computer-based Scenario Simulations
Computer-based Horizontal simulations
Computer-based Vertical simulations
Computer-based full functionality simulations
First article build
High-Fidelity
SYS 511 – Ver 2.1
51
Summarizing Lo-Fi versus Hi-Fi
Type
Advantages
Low – Fidelity
•Less Time & Lower Cost
•Can evaluate multiple concepts.
•Communication tool.
•Good for User Interface
development
•Limited usefulness for usability
tests.
•Navigation and flow limitations.
•Facilitator-Driven
•Poor for detailed specs.
High Fidelity
•Partial / Complete Functionality
•Interactive.
•User-Driven
•Can clearly define navigational
schemes.
•Good for exploration and test.
•Marketing and sales tool.
•Time-consuming
•Inefficient for proof-of concept
issues.
•May bind users to representation
short-comings.
•Management (customer) may think
it is real.
SYS 511 – Ver 2.1
Disadvantages
Considerations when choosing
 Cost and Schedule Constraints.
 Lo-Fi – cheap and fast.
 Hi-Fi – slow but sure.
 Need for “Proof-of-concept.”
 Mission Needs Statement Validation
 Often Lo-Fi and Horizontal.
 Navigation and Flow issues
 Medium to Hi–Fi prototyping to simulate interactions, possibly in system
“real-time”.
 User-driven versus Facilitator-driven
 User-driven – may need to be Hi-Fi.
 Facilitator Driven – may get by with Lo-Fi.
SYS 511 – Ver 2.1
53
Considerations when choosing (Cont’d)
 Need for “Look and Feel”
 Usability Testing
 Hi-Fi needed to provide realistic appearance and functionality.
 Design Team experience
 High skills and relevant experience – Lo-Fi.
 New team or new concepts – Hi-Fi.
 Need for a “Controlled Study”
 Lo-Fi may not provide the stimulation needed.
 Special Equipment Needed?
 Stereo lithography (or similar)
 Special Software
 Special Personnel Needed?
 Videographer
 Representative user
SYS 511 – Ver 2.1
54
The Perils of Prototyping
 Standardization – Prototypes tend to be shaped by the tools that are
available rather than the user’s needs.
 Distraction – Work on the prototype can take attention away from the
problems to be solved.
 Seduction – Developers can be trapped in an endless loop of
refinement.
 Rejection – If the cost of implementing (prototyping) an idea is too
high, ideas may be rejected too early in the cycle.
 Obscured historical perspective – Prototypes tend to lose the
reasoning that went into them.
SYS 511 – Ver 2.1
55
Distributed Interactive Simulation (DIS)
 Distributed Interactive Simulation (DIS) is an IEEE standard (IEEE 1278)
for conducting real-time platform-level simulations across multiple
host computers but has been adopted by other domains
 Purpose to create realistic, co…
Purchase answer to see full
attachment