Having worked using old-school workflows like Gitflow or feature branches, it happened that sometimes, the code I wrote was finally visible to the end user months after the fact.

At some points, the following questions started to haunt me:

• "How long does it take to see your code changes in production"?

• "How fast can you get feedback from the users"?

• "How fast can you revert changes when a critical bug occurs"?
  And the list goes on.

So what could I do to improve the situation and not feel frustrated continuously?

The answer was: shifting workflow and using the right tool for the right job. By shifting from a "feature-based" approach to a more "code-based" approach, it became possible to deploy code multiple times a day. And this is the beauty: it's okay to deploy small pieces of code even if the associated feature is not finished yet.

We deploy code, but we release features. The idea is to deploy code many times until it finally creates a whole running feature.

"But wait, how do you accept showing unfinished features to your users?!"

You don't, you guard your unfinished feature with feature flags.

Using feature flags in a codebase is as simple as adding a conditional statement. For context, people have been using environment variables for this use case for years.

Let's think of it for a second: using environment variables means that when you want to deploy your feature, you need to rebuild it with the new environment variable value and to re-deploy it. Depending on your application it may take time.

What if instead, the value of the condition came from a remote service at runtime? It means that its value could change over time without the need for a whole build and deployment of the application.

Let's push this even further: since the condition is on a value that can change over time without re-deploying, it opens the door for a lot of great features:

• Rollout / Rollback at any moment

• Time-based rollout (activate the feature at a given date)

• Progressive rollout (only a % of the audience has access to the feature)

• Field-based rollout (using email, id, etc..., practical to create Canary & Beta user groups)

• A/B testing (a feature flag can have multiple values and so)

• A huge range of analytics and AI application for understanding the audience activity

This is why Progressively exists. Make sure to give it a try and give feedbacks :)

Whether you have an already running application or you're about to start one, there is a bunch of things to accomplish before being fully operational.

This blog post will deal with databases and backend that you may not have if you run on static websites. However, most of the content will be applicable in your case anyways.

Establish good foundations

Define what is E2E testing as a team

Depending on your overall ecosystem, your team or your project, make sure that everybody has the same definition of E2E testing. I'm saying this because I've seen many people with their definitions of testing automation, but from one person to another, they differ drastically. Make sure everybody is on the same page before starting.

Define a static dataset

This is probably the most important thing to keep in mind: for your tests to be deterministic, you will need to have a predefined set of data on which you can make expectations that will guarantee the same result.

You don't need an entire dataset, with all the relationships taken into account, nor try to create an exhaustive one for an app running on top of 2k SQL tables.

You don't need everything to be ready from the ground, start small, write your first meaningful tests and iterate from there.

If you have a complex application with multiple personas or different access rights, make sure to have predefined accounts for each of them.

Create shortcuts for seeding & resetting the database(s)

Since you have a dataset that is ready to be used, make sure to create functions or custom commands that will put the dataset into the database, and also a command to erase the data from the database.

This will be practical to ensure a coherent state between your test runs.

One positive effect of these functions/commands is that if you have a newcomer in your team, they have a predefined set of users to play with.

Create shortcuts for authenticating your users

You are about to test your applications, and chances are that you will need to be authenticated to perform some actions.

I would highly recommend creating a dedicated function for this specific purpose: make sure it can accept a login, a password or whatever is required to authenticate your user.

Also, be aware that E2E tests take time since they require your whole stack to be started: if you can skip browsing the login page with your E2E test tool and directly get a token from your API with a fetch call, do it, it will speed up your tests run by a significant amount.

Writing meaningful tests

Test like a real user effectively

People tend to think that E2E tests should test a whole functionality: I agree and disagree. It depends on the size and number of actions you have to make to perform a specific action, and also on the way you deal with the state of your application in your frontend (state by URL vs in-memory a-la Redux, Zustand and friends...).

I find E2E tests to shine when I focus on a given page instead of focusing on a whole feature itself for the following reasons:

• Easier file system organisation: 1 test file, 1 route

• You don't test browser APIs (link navigating between pages with <a> tags for instance)

• Less cognitive load, focus on the current page, the scope is clear

• Easy to shoot accessibility tests per page

It's opinionated, but this is the way I have most enjoyed these last 10 years.

Test everything in isolation

Make sure that you are in a consistent state between the different test runs. Be sure that your database has good data in good shape to make a meaningful assertion. If you don't pay too much attention, you might enter situations where your tests will be flaky and your expectations may fail sometimes, but pass other times.

One good rule of thumb is to seed and reset your database between tests.

At some point, you will face the problem that we all have had when writing E2E tests: they are slow. And you will feel the urge to not seed and reset your database on every test run. If this is the case, make sure that you test behaviours that can't collide.

Keep tests simple

The easier it is to read, the easier it is to debug. If a new person joins your team, they should be able to jump on the train without friction.

Keep in mind that for many people, E2E tests are hard. Don't over-abstract, or create unnecessary function indirections for the sake of smartness, keep things simple and scalable. Duplicating code is fine, your colleagues will thank you.

Query your elements by their role

When writing E2E tests, you make DOM queries to select specific elements and you make assertions on them. For example, you can verify that the title is well set, that when you press that specific button the counter increments etc.

There are debates on how you should query elements on which you want to perform actions or assertions (query by id, by text etc...): my go-to would be to query the element by its role + its content. The positive effect of that is that even if you don't have experience with accessibility, you are already one foot in by making sure you are clicking on a "button" and not a "div" (please, stop putting click events on divs).

Make them part of your workflow

Setup your CI

Remember that an E2E test is about spawning your whole application, from one end to the other. You have to start the database(s), the frontend(s) and the backend(s).

If you use Github Actions, I have an example for https://progressively.app in this file.

Once everything is setup, you can effectively start your E2E tool and play the tests in headless mode.

Dealing with speed issues

You will face speed issues when running E2E tests in your CI. The first reason is the setup part: you need everything to be up and running and it takes time.

The second reason is that you need your tests to run sequentially on a given machine. You can't parallelize E2E tests on one machine: if you do that, you will encounter an inconsistent state because your data stored in the databases will not be what you were expecting. What if one test tries to assert while the other is currently resetting the database? The test can't pass.

One way to solve this is to run your E2E tests on multiple machines. Group your tests by running speed (don't put two long-running suites in the same group) and spawn your whole environment multiple times on multiple machines. For instance, instead of running only against 1 database, you can now run against 4, seed and reset them independently.

If using Github actions, you can tweak the `matrix` option to achieve this.

In this post, I will share with you my vision of A/B testing (or feature flagging, everything written below may apply to both) using the JAMstack.

But first, why do we use the JAMstack?

According to jamstack.org, we use it because:

• it's about building simple static pages (HTML files)

• most of the job is made at build-time, no potential leak at runtime, it's more secure

• static pages are HTML files and thus load very fast

• we benefit from cheap hosting (putting files on a server and there we go)

• it's highly scalable, just put the files on another machine and scaling is done

• it already exists great tools to create amazing sites (Gatsbyjs, Nextjs, 11ty, etc...)

Why do we use A/B testing?

People use A/B testing to measure which variant of a website is more appreciated by their users. The idea is simply to provide different visual representations of some data and check which one attracts more people.

If the visitors of variant A come more often than the visitors of variant B, then we can assume that variant A is a more viable solution to represent the data on the page.

How do we use A/B testing in non-JAMstack applications?

In applications that are not built on top of the JAMstack, the idea is quite straightforward.

When opening the application, it will request a remote server to get the different available variants. Then, based on some conditions in the codebase, we can display the good variant to the good user.

The following is an example of a client-side A/B test written with Reactjs:

const App = () => {
  const { variant, isLoading } = useABVariant();

  if (isLoading) {
    return <p>Preparing the application...</p>;
  }

  if (variant === "A") {
    return <div>Here's the A variant!</div>;
  }

  return <div>Here's the B variant!</div>;
};

As we see in the snippet, the code is executed in the user's browser. Also, notice the loading information while the request is pending before being able to display the variant content.

The following is a tiny sandbox allowing to switch on and off a feature flag to display a new variant of a homepage. When using services like LaunchDarkly, this is exactly the kind of actions you're provided with.

Why A/B testing on the JAMstack is different?

Remember one of the main arguments for building on top of the JAMstack is fast page loading (performance).

When dealing with A/B tests the "standard way", we need to make an HTTP request to get the different variants. Making an HTTP request means that there is a delay between the moment we ask for the variants and the moment we get them back from the server.

The problem is that making the HTTP request is so critical that we can't show anything else than loading information to the user before resolving the variants and being able to show them the good content.

In a static environment, we are waiting for dynamic information to display meaningful information.

When A/B testing the "standard way", using runtime information will make the application performance worse by increasing the time for displaying the first meaningful content. Loading static pages should be "instant" but relying on an HTTP request and an intermediate loading state before displaying the content will take extra time and decrease the experience.

In some scenarios, Lighthouse performance score can drop by around 25 points (up to you to determine if it's significant or not).

Also note that some tools helping build applications using the JAMstack don't even run JavaScript at all, meaning that it's not possible to rely on HTTP requests to access remote data at runtime.

How to make A/B testing in a JAMstack fashion?

The JAMstack is about building static pages. Taking this notion to the extreme, we can imagine creating a dedicated set of static pages for different variants and host them in different places, like for example, different machines.

Machine 1 owns all the statically generated HTML pages impacted by the variant A and machine 2 owns all of the statically generated HTML pages of the variant B.

Since pages are statically generated at build time, we can rely on environment variables to display the good variant content:

const AB_EXPERIMENT_VARIANT = process.env.AB_EXPERIMENT_VARIANTS;

const App = () => {
  if (AB_EXPERIMENT_VARIANT === "A") {
    return <div>Here's the A variant!</div>;
  }

  return <div>Here's the B variant!</div>;
};

The next step is to rely on some kind of proxy to route the different users to one of the two variants and make sure they always see that variant.

Remember, we can't rely on runtime information to store the variant, like an authenticated user id for example. We need to rely on something else. Hopefully, HTTP Cookies exist that allow for a client-server kind of data sharing. We can benefit from them to store the actual variant requested by the user and make sure that they will always get routed to that variant.

Observations on this approach

The first observation to push upfront is that we have kept the page loading performances provided by default by the JAMstack. We don't have to wait for runtime computations to display content on the page.

The second observation is about the tradeoffs we decided to make: we shifted the A/B testing responsibility closer to the infrastructure layer. The proxy has an important role to play in this scenario.

Also note that we need more "places" to put the different variants: 1 variant corresponds to 1 website that corresponds to 1 set of static pages that should entirely be hosted. The more we have variants, the more we may pay for hosting:

There's also one side effect that I find positive (but it can feel negative) is that it can be hard to combine multiple A/B experiments.

Let's say that we have 2 pages and that we want to run an experiment on both pages:

• What can be a visual representation of these scenarios at the hosting level?

• Should we create 4 different websites, one for each variant?

• Should we create N different websites with combinations of variants?

My expectations are that there will be solutions allowing to make A/B testing at the file level itself, and not at that site level.

References

If you're using Netlify for hosting your websites, they have a feature called Split-testing that allows for this kind of testing using a branch-based approach.

If you have any other references in mind concerning A/B testing or Split Testing on top of the JAMstack, feel free to drop them on Twitter.

Don't forget that the goal of a developer is to ship value and Gitflow/feature branches were slowing me down in this regard.

With trunk-based development, I create very small branches that live for a few hours, and each time I merge one in the main branch, the code is deployed to production. To protect users from accessing unfinished features, I use feature flags: each time I want to activate a feature, I just press a button.

How fast is that?

Here are 10 things about feature flags that will speed up your workflow.

1. Separate code deployment from feature releases

You can deploy code 20 times a day, and all this code may not act as a fully polished feature. And it's okay because with feature flags, activating a feature to the audience is not tightly bound to your code deployment process: it's just a toggle button that you click in a dashboard to make the feature available to the users.

if (flags.newFeature) {
  return <NewFeature />
}

return <OldFeature />

2. Rollbacks are easy

Almost the same point as above: since the code deployed and the feature are not strictly bound, you can deactivate the feature with just one click and not go through your whole CI/CD pipelines to get back to a valid and running state.

3. Test directly in production

In most of the feature flags tools available in the market, you can decide which users can see the new feature and which users can't. It means that you can activate the feature only for your QA team and they will be able to play directly in production.

Side effect: no discrepancies between environments anymore, it's always the same server configurations.

4. Unlock Canary and Beta releases

Using the same targeting tooling as previously explained, you can create a Canary users group and a Beta users group and activate the feature very early for them.

5. Gradual rollout

Let's say you've built a brand new feature, and you don't feel very confident about what you've created. With feature flagging tools, you can increasingly roll out your feature to 1%, 2%, 3%...100% of your audience.

The funny thing is that you can correlate the percentage of rollout with your error tracking tool (such as Sentry) and check if your new feature has increased your error rates.

6. The A/B testing side effect

Feature flags are generally used with "true" and "false" values: is the feature activated or not?

By pushing the concept one step further, we can give feature flags multiple values "A", "B", "C" (or whatever) and display different UI/UX of a given feature to a percentage of the audience. You can then measure the number of actions taken on each of these different views and see which one converts the most.

7. Time-based rollout

Do you have something to deploy on Monday at 9 am? With feature flagging tools, you can automate that. Since, again, the feature and the code are not tightly bound, the code already exists in production: all you need to do is to create a rule in the dashboard of your tool and at 9 am Monday, you'll see your new feature live.

8. No impact on web performance with Progressively

Progressively is a feature flagging tool and in contrary to other solutions, the client-side SDK, which works in real-time, is only 1.3kB.

Stop fearing the slow page loading because of third-party tools anymore.

9. Progressively with SSR and Edge Runtimes

The tool works with SSR frameworks such as Svelte, Qwik, Astro, Nextjs, Remix etc... You don't need to wait for the client side of your app to load and the page to flicker to benefit from a great experience with feature flags.

And if you want to work at an even bigger scale, try to use Progressively inside an Edge Middleware in Vercel ;).

10. Progressively is FREE

It's open source, and you can self-host it on your servers. Go to https://github.com/progressively-crew/progressively, follow the instruction and spawn your instance on your machine, for free.

And if you're not so much on self-hosting, there is a SaaS solution that you can pay to play with.