From f1f79ee85f3488a682dfb5bedb5a779365759bed Mon Sep 17 00:00:00 2001 From: Florian Schmidt Date: Fri, 15 Jun 2018 11:13:39 +0200 Subject: [PATCH] [FLINK-9594][E2E tests] Improve docs for new test runner This closes #6172. --- flink-end-to-end-tests/README.md | 42 +++++++++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 4 deletions(-) diff --git a/flink-end-to-end-tests/README.md b/flink-end-to-end-tests/README.md index 82d7e419ad8..26dba3d5dd3 100644 --- a/flink-end-to-end-tests/README.md +++ b/flink-end-to-end-tests/README.md @@ -28,11 +28,45 @@ where is a Flink distribution directory. You can also run tests individually via ``` -$ FLINK_DIR= flink-end-to-end-tests/test-scripts/test_batch_wordcount.sh +$ FLINK_DIR= flink-end-to-end-tests/run-single-test.sh your_test.sh arg1 arg2 ``` ## Writing Tests -Have a look at test_batch_wordcount.sh for a very basic test and -test_streaming_kafka010.sh for a more involved example. Whenever possible, try -to put new functionality in common.sh so that it can be reused by other tests. +### Examples +Have a look at `test_batch_wordcount.sh` for a very basic test and +`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try +to put new functionality in `common.sh` so that it can be reused by other tests. + +### Adding a test case +In order to add a new test case you need add it to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh`. Templates on how to add tests can be found in those respective files. + +_Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually._ + +### Passing your test +A test is considered to have passed if it: +- has exit code 0 +- there are no non-empty .out files (nothing was written to stdout / stderr by your Flink program) +- there are no exceptions in the log files +- there are no errors in the log files + +_Note: There is a whitelist for exceptions and errors that do not lead to failure, which can be found in the `check_logs_for_errors` and `check_logs_for_exceptions` in `test-scripts/common.sh`._ + +Please note that a previously supported pattern where you could assign a value the global variable `PASS` to have your tests fail **is not supported anymore**. + +### Cleanup +The test runner performs a cleanup after each test case, which includes: +- Stopping the cluster +- Killing all task and job managers +- Reverting config to default (if changed before) +- Cleaning up log and temp directories + +In some cases your test is required to do to some *additional* cleanup, for example shutting down external systems like Kafka or Elasticsearch. In this case it is a common pattern to trap a `test_cleanup` function to `EXIT` like this: + +```sh +function test_cleanup { + # do your custom cleanup here +} + +trap test_cleanup EXIT +```