July 2023 Engine Update
Learn how the new engine helps with previous limitations and what are the trade-offs.
Engine Update Timeline
We will be releasing the new version of the Scenarios generation engine by the end of July. Upon release, both engines will be available side-by-side to give you time to transition. This project should have no impact on how the current engine works.
We will be retiring the old engine by the end of August.
How to activate the new engine
Overall, you will be performing the same steps you normally would to generate the scenarios table.
You will now see 2 options under the Scenarios section: “Original” represents the old engine, “Standard” – the new one. As mentioned, you can use both initially, but we will eventually transition to only supporting the “Standard” version.
What are the advantages of the new engine
- Improved Speed; larger models with a lot of constraints will show larger improvements.
- In larger models and/or at higher coverage strengths, you will see scenario rows being “streamed” in real time (instead of waiting for the entire table to finish before rendering)
- In some instances, you will also seem improved value variety across rows (even if the incremental coverage % stays the same)
- Full rendering; there will never be “any valid values” present in the generated scenarios.
Improved Constraint Handling; “No possible value” as the table element will never appear with the new engine.
There were 2 types of NPVs:
- user error in constraints;
- complex logic (especially involving Skip constraints) that actually had viable values available in the scenario slot, but the old engine couldn’t figure it out.
The new engine solves the second issue automatically. For the first one, there will be a warning that certain values cannot be used in the test suite and the Scenarios table won’t generate.
What are the trade-offs of the new engine
- Constraints and Forced Interactions alignment
As the consequence of improved constraint handling, it is no longer possible to have constraint vs forced interaction contradictions (previously doable when accepting the warning). That means certain implementations of one-off scenarios or n-way constraints are not supported by the new engine.
We are working on a dedicated option for n-way constraints and evaluating other approaches for one-off tests.
- (IMPORTANT) Non-determinism
As you will notice, clicking the “Scenarios”->”Standard” link multiple times throughout the modeling process can result in different content and counts of scenarios even when it seems counterintuitive. For example,
- Copying a model can lead to a lower or higher count;
- Adding a value to the largest parameter can lead to a lower scenario count;
- Adding a value to the smallest parameter can lead to a higher scenario count;
- Removing a value can lead to a higher scenario count;
- Replacing a value can lead to a lower or higher count.
That is because the new algorithm searches for the approximate optimum and uses some non-deterministic variation in the process. In other words, the engine “knows” that the optimal number is in the ballpark of X, but doesn’t prioritize always returning exactly X (unlike the old engine) – hence the trade-off between accuracy/consistency and speed/other benefits mentioned above.
Therefore, the scenarios chosen for each iteration will be slightly unique – e.g., if you run the uncached (see below) generation 10 times, you can have 10 different outcomes – even if the model shape remains more or less the same. The count difference in these counterintuitive instances is usually within 5.
The only (guaranteed) unchanging scenarios are those in the cache, and any change to the model shape will break the cache. For example, the following will force a “fresh” generation:
- Adding/Removing parameters
- Adding/Removing values
- Adding/Removing/Editing constraints
- Adding/Removing/Editing forced interactions
- Copying the model/reverting to a different version of the model
The examples of model actions that do not break cache:
- Editing value expansions
- Renaming values
- Navigating between tabs
- Adding notes
- Editing scripts
Importantly, just switching the computation strength within the same engine (e.g. from 2-way to 3-way) does not remove the 2-way cache.
Equally importantly (for the transition period), switching between engines will wipe out the cache (so, going Standard -> Original -> Standard can have new results/count).
A couple of aspects to keep in mind:
No redundancy – despite the count differences, there will be no duplicate scenarios unless you intentionally include them via forced interactions. The delta will just tap into higher coverage strengths.
Freezing – once you are happy with the set of scenarios, if you want to maintain consistency for execution, we recommend using the “Freeze these Scenarios” option in the top-right of the Scenarios screen. That will make sure that any changes due to model updates and non-determinism are happening at the bottom of the table, to newly added scenarios, where they can be more easily incorporated or ignored.
Keep in mind that Freezing does not affect the functions built on top of values (e.g. value expansions, ranges). Those elements may show different choices when a frozen set of scenarios goes through regeneration. The more value expansions there are and/or the wider ranges are, the more likely you will see discrepancies.
Analysis, Scripting, and Export – the Coverage Matrix/Graph and scripting/export options will adjust to the latest scenarios count & content for the selected strength.
Approximate Optimum – we plan to introduce an “Optimized” generation option which will use 50 times the computing power to strive towards a more optimal solution. This will often result in slightly reduced scenario counts. The non-deterministic variation is critical to making this optimization possible.