At around 11.55 PM AEST, 30/08/2024, a team member informed us that several clients were experiencing issues with creating new content and managing permissions in Iris.
Senior engineering staff in the EU investigated the issue and decided that it would be addressed by members of the engineering team in Australia on Monday September 2nd.
This further investigation took place at 9.00 AM AEST 2/09/2024. During the investigation, we uncovered the following:
- Upgrading the segmentation UI widget that Iris uses inadvertently triggered an update to a third part library in Iris.
- This updated library was not compatible with another third party library. However, this was not evident as the build succeeded when it should have failed. We discovered after the fact, that there is a gap in our automated tests in this area.
- The Iris UI application has a non-standard deployment process, and is deployed by a manual trigger of the build/deployment process. This is inconsistent with the rest of our platform.
- A change was later made to the Iris back end on Friday and this was manually deployed when the build was successful. This inadvertently deployed the Iris UI build with the updated segmentation UI
The above incident resulted in the following impacts:
- Users were unable to create new content in Iris, or update users permissions..
As a result of this incident, we will
- Increase automated test coverage in the areas we discovered gaps
- Mandate the versions of specific third party libraries in Iris, so any update to these packages are deliberate choices.
- Increasing regressions tests when third party libraries are updated
- Update Iris packages or replace Iris
- Update Iris to deploy automatically on successful builds - to bring it in line with the rest of our platform