Building Preview Environments

Preview or ephemeral environments are one of the most useful DevOps CI/CD features that you can add to your team's capabilities. As agile and DevOps main goals have always been to enable fast feedback and experimentation.

In this article, we will guide you through: what is preview environment is and important implementation considerations. Based on our last 7 years of experience building such a feature

What Are Preview Environments?

Preview environments or ephemeral environments (PE) are on-demand self-service created application versions. PE's are one of the most valuable features of CI/CD DevOps platform offerings. This is due to the following reasons:

Faster feedback cycle as software features or bug fixes can be demonstrated and tested in an integrated environment;
Reduced congestion comparing to shared environments as most unit/integration/acceptance testing can be done in separate environment;
Better overall reliability and resilience for infrastructure, as environment setup is part of the daily routine, not just an exception. Assuming that the same configuration is being used to provision PE's and standard environments. And corrective measures were applied

Preview Environment Design Considerations

We have been implementing preview environment features for more than 6 years. During this time, processes have become much easier, but still there are considerations that need to be addressed to archive the best results

Buy vs Build

When we started implementing PE, it was a desirable but exotic feature. Now it's a pretty common feature for many CI/CD vendors. However, in most cases, those who need PE's have some build infrastructure in place. GenAi have, significantly, reduced costs for "build" option. Our recommendations: - When starting fresh or rebuilding CI/CD infrastructure, consider PE feature requirement. This is not a nice-to-have feature as it brings significant value for the organization; - Don't extract PE workflows from common CI/CD infrastructure in a separate product/platform, without integrating the developer workflow. - If starting from scratch, build is preferable, but if existing infra exists, build/extend. Of course, each case is different.

Preview Environment Scoping

PE scope will define what components to include, the complexity of deployment, and what internal and external dependencies will be included. Two important things need to be considered when doing scoping for PE:

What is the boundary for PE, and what components will be included in the preview environment? Partial components or a fully working tenant. Our recommendation is that, PE scoping should reflect the development team scope of work, however, our prefference is to build as fully working as possible tenant.
If the preview environment scope is just part of the full tenancy, what integration endpoints to used. And how hard is it to change these endpoints if needed for individual PE. Our recommendation is to make it possible to configure endpoints for CI/CD platform for developers. However, data and general security need to be considered. In order not to don't break production and not to cause security threats.

Environment scoping goes very much with domain-driven desing and microsevice architectures. There are a lot of good resources, but this is the good reference point.

Preview Environment Data and Configuration Provisioning

Phill Carton once said: "There are only two hard things in Computer Science: cache invalidation and naming things." We can rephrase this quote for preview environments: "There are two hard things in PE: data provisioning and naming things". PE is relatively easy when stateless systems are sed.

However, almost all significant systems (for exampl,e invoicing software) consist of 3 types of data:

Datastore metadata, like databases, schema, tables, triggers, indexes, etc.
Product configuration data, like settings and secrets
Application test data, like example clients, example registrations, and transactions

We can also look complexity perspective, how to provision these 3 artifact types:

Data Type	Complexity To Generate	How to Generate a Data Type
Datastore metadata	Easy to Medium	Schema migration tools, like Flyway or Liquibase; DB dumps;
Product Configuration	Medium	Configuration manifests, like YAML; Schema migration tools; DB dumps
Application test data	Complex	DB dumps, Schema migration tools Custom tooling

Our recommendations:

When you start to implement PE, start from this area. This is gonna be the most complex aspect. As it will involve: developers, the DevOps side, and the security/compliance team. To comply with data protection requirements.
We have found. That product configuration can be characterized in two main blocks: product infrastructure configuration (PIC: like database connection strings) and product business configuration (PBC: like credit checks and limits ). PIC should be a minimal number of configs to start the application preview environment. PBC should be safe defaults, with the ability to change.
Investments in application test data generation techniques. Like use-case test scenarios where simulation of actions is possible (for example: issuing a credit card process). In mid/long term, such approaches bring high yield returns, as it's possible to build proper acceptance testing;

Workflow To Create Preview Environments

When and what is triggered to create preview environments will impact the developer experience and usefulness of the feature. The main rule of thumb, it should be as convenient as possible for the development team and reflect how they are working with code.

There are two main schools of thought for provisioning:

Automatic provision trigger: most cases within MR/PR when a team works with pull requests as a code review tool;
Manual provision trigger: cases when pull requests are not used
On custom condition triggers, like tags or comments in a commit

Our recommendations:

In order to gain PE popularity, it has to be easy to provision and integrated into the existing CI/CD process.
Good documentation of principles and getting started is crucial for success;
When the environment is created, it should be rebuilt on each push to the source branch to stay up to date. Explore ways, such as caching, to reduce build cycle time.
Always include in provision summary info about how to access the built environment, like URLs/credentials

Disprovisioning Preview Environments

Preview environment design considerations must also include how to deprovision them. As PE's are not meant to be long-running and generate running costs.

Common approaches to de-provision:

On Time-based: environments expire after X days. Either from creation or the last update.
On branch closure: when fa eature branch or pull request is merged
On custom conditions, like comments in the commit message

Our recommendations:

Document and explain principles and conditions when environments are shut down
Document and explain how to recover the expired environment

Conclusion

To summarize. PE's are designed to test individual features in production, like setting up faster, avoiding environment congestion and and improving the reliability of the general system. The feature is valuable, but comes with some design considerations, what was addressed here.