mostlymatter/e2e-tests/README.md

# E2E testing for the Mattermost web client

This directory contains the E2E testing code for the Mattermost web client.

### How to run locally

##### For test case development

Please refer to the [dedicated developer documentation](https://developers.mattermost.com/contribute/more-info/webapp/e2e-testing/) for instructions.

##### For pipeline debugging

The E2E testing pipeline's scripts depend on the following tools being installed on your system: `docker`, `docker-compose`, `make`, `git`, `jq`, and some common utilities (`coreutils`, `findutils`, `bash`, `awk`, `sed`, `grep`)

Instructions, tl;dr: create a local branch with your E2E test changes, then open a PR to the `mattermost-server` repo targeting the `master` branch (so that CI will produce the image that docker-compose needs), then run `make` in this directory.

Instructions, detailed:
1. (optional, undefined variables are set to sane defaults) Create the `.ci/env` file, and populate it with the variables you need out of the following list:
  * `SERVER`: either `self-hosted` (default) or `cloud`.
  * `CWS_URL` (mandatory when `SERVER=cloud`, only used in such caase): when spinning up a cloud-like test server that communicates with a test instance of customer web server.
  * `TEST`: either `cypress` (default), `playwright`, or `none` (to avoid creating the cypress/playwright sidecar containers, e.g. if you only want to launch a server instance)
  * `ENABLED_DOCKER_SERVICES`: a space-separated list of services to start alongside the server. Default to `postgres inbucket`, for smoke tests purposes and for lightweight and faster start up time. Depending on the test requirement being worked on, you may want to override as needed, as such:
    - Cypress full tests require all services to be running: `postgres inbucket minio openldap elasticsearch keycloak`.
    - Cypress smoke tests require only the following: `postgres inbucket`.
    - Playwright full tests require only the folliwing: `postgres inbucket`.
  * The following variables, which will be passed over to the server container: `MM_LICENSE` (no enterprise features will be available if this is unset; required when `SERVER=cloud`), and the exploded `MM_ENV` (a comma-separated list of env var specifications)
  * The following variables, which will be passed over to the cypress container: `BRANCH`, `BUILD_ID`, `CI_BASE_URL`, `BROWSER`, `AUTOMATION_DASHBOARD_URL` and `AUTOMATION_DASHBOARD_TOKEN`
  * The `SERVER_IMAGE` variable can also be set, if you want to select a custom mattermost-server image. If not specified, the value of the `SERVER_IMAGE_DEFAULT` variable defined in file `.ci/.e2erc` is used.
  * The `TEST_FILTER` variable can also be set, to customize which tests you want cypress to run. If not specified, only the smoke tests will run. Please check the `e2e-tests/cypress/run_tests.js` file for details about its format.
2. (optional) `make start-dashboard`: start the automation-dashboard in the background
  * This also sets the `AUTOMATION_DASHBOARD_URL` and `AUTOMATION_DASHBOARD_TOKEN` variables for the cypress container
  * Note that if you run the dashboard locally, but also specify other `AUTOMATION_DASHBOARD_*` variables in your `.ci/env` file, the latter variables will take precedence.
  * The dashboard is used for orchestrating specs with parallel test run and is typically used in CI.
  * Only Cypress is currently using the dashboard; Playwright is not.
3. `make`: start and prepare the server, then run the Cypress smoke tests
  * You can track the progress of the run in the `http://localhost:4000/cycles` dashboard, if you launched it locally
  * When running with `SERVER=cloud`, this will automatically create a cloud customer against the specified `CWS_URL` service, and delete that user after the run is complete.
  * If you want to run the Playwright tests instead of the Cypress ones, you can run `TEST=playwright make`
  * If you just want to run a local server instance, without any further testing, you can run `TEST=none make`
4. `make stop`: tears down the server (and the dashboard, if running)
  * `make clean` will also remove any generated environment or docker-compose files, in addition to stopping the containers.

Notes:
- Setting a variable in `.ci/env` is functionally equivalent to exporting variables in your current shell's environment, before invoking the makefile.
- The `.ci/.env.*` files are auto generated by the pipeline scripts, and aren't meant to be modified manually. The only file you should edit to control the containers' environment is `.ci/env`, as specified in the instructions above.
- Aside from some exceptions (e.g. `TEST_FILTER`), most of the variables in `.ci/env` must be set before the `make generate-server` command is run. Modifying that file afterwards has no effect, because the containers' env files are generated in that step.
- If you restart the dashboard at any point, you must also restart the server containers, so that it picks up the new IP of the dashboard from the newly generated `.env.dashboard` file
- If new variables need to be passed to any of the containers:
  * If their value is fixed (e.g. a static server configuration), these may be simply added to the `docker_compose_generator.sh` file, to the appropriate container.
  * If you need to introduce variables that you want to control from `.ci/env`: you need to update the scripts under the `.ci/` dir, and configure them to write the new variables' values over to the appropriate `.env.*` file. In particular, avoid defining variables that depend on other variables within the docker-compose override files: this is to ensure uniformity in their availability, and simplifies the question of what container has access to which variable considerably.

##### For code changes:
* `make fmt-ci` to format and check yaml files and shell scripts.