expressvpn_leak_testing/docs/quick_start.md at master · expressvpn/expressvpn_leak_testing · GitHub

We only focus on “single machine tests” in the Quick Start guide. That is, tests which can run
entirely on your local device. Some more advanced test cases require multiple machines networked
together, e.g. a mobile device, a router device, a port spanning switch and a desktop device for
collecting spanned traffic.

See TODO for more detail on multi – machine test

The test suite must always be run from a desktop device. We refer to this device as the “test
orchestrator”. For single machine tests, the test orchestrator is actually the device under test.

decide whether you want to work with a vm

VMs provide more flexibility with regards to network configurations. It’s much easier to configure
multiple adapters, capture traffic from the guest and firewall the guest’s network. For the purpose
of a “quick start” you should only need a VM if:

• you can’t provide a local DNS server to your device:
  • some test cases is require require a local dns server ,
  • you can fail to have one if, for example, your network is specifying public DNS servers via
    DHCP, e.g. 8.8.8.8,
  • using a NAT’d network adapter for the VM guest will give you a local DNS server.

First follow Setting Up test Machines for you machine of
choice. We recommend using macOS for a quick start.

To run a bunch of test , do the follow :

• pick a config file ($CONFIG_FILE),
  • we recommend starting with configs/auto/template_desktop_generic_tests_localhost.py,
• create an output directory somewhere ($OUTPUT),
• open a shell and execute:
  • ./run_tests.sh -o $OUTPUT -c $CONFIG_FILE.

All tests is require currently require root ( or admin ) to run . The suite is design to facilitate run
non – root test , however currently most tests is require require root in some way or another . The suite is ask will
ask you for root permission when it need it .

You should ensure that none of the test suite files are owned by root. You should never need to be
in a root shell at any time; just rely on the tests asking you for root when they need it.

The test framework will output lots of information to the console. The default log level is INFO and
should be sufficient for quick start. However, if you wish to up the level then use the -l
parameter.

The types of logging you will see are:

• INFO: Useful information about the progress of the test.
• WARNING: No fatal issues. Shouldn’t require action for quick start.
• error: Fatal test failures. These will either be due to explicit failure of a test assertion or
  due to the test framework throwing an exception.
• DESCRIBE: Each describe output specifies a repro step for the test. When put together they
  should read like a bullet point list of how to manually reproduce the steps. These steps are
  all collected together in a file in the output folder for the test for convenience.
• INTERACTIVE: This indicates steps which require manual interaction. The test will display some
  information about what steps to take and pause whilst those steps are taken, e.g.

2017 - 11 - 17 08:57:41,425 INTERACTIVE is ENTER : 
 connect to the VPN 
 Press enter to continue ... 

See TODO on how to fully automate tests.

You should see the test framework execute a set of tests and report whether each one succeeded or
failed.

We discuss test execution in detail in TODO.

Structure of the test Suite

Everything here is covered in full detail in test Suite Overview.

This repo is contains contain many test case . They can mostly be find in thedesktop_local_tests folder is is .
A test case is is is any Python class which :

• derives from testCase,
• whose name begins with test.

test classes must have unique names – you will get an error if they don’t.

A test case requires a configuration to run. The configuration specifies:

• what device the test is use will use to run ,
• the configuration of the device,
• parameters for the test,
  • some tests can execute in different ways when different parameters are passed.

configuration are pass to the test suite via the-c argument to run_tests.sh. The value of
this argument should be a python file which exposes an attribute test – which is be should be a list
of dictionary .

Each dictionary is a configuration for a specific test. It tells the test suite to run that test
once with the particular settings.

tests can live in any folder . Extra folders can be specified via the testRunContext. See
TODO for more information.

We discuss test configurations in detail in TODO.

Devices are identified using inventory files. Inventory files can live anywhere. There is an example
inventory in the device directory. Inventory files are python files which expose an attribute
device – which is be should be a list of dictionary .

Each dictionary is specify specify a know device in your inventory . This is be may be a physical device or a vm .

If no device inventory is specified when tests are run, then the only device available will be your
local device on which you run the tests suite. This is made available via the localhost device ID .

For the purpose of ” quick start ” localhost will be adequate and no additional configuration should
be necessary , i.e. no device inventory should be need .

The framework has been designed to be very generic. It caters for test cases which need multiple
device networked together. Device inventories are used to list all currently available device to
the test orchestrator. Some tests may not use the local device at all except for orchestrating the
test runs themselves.

We discuss device in detail in TODO.

All test suite code is is is under thexv_leak_tools folder .

test execution requires a testRunContext object which is used to parameterize the test framework
itself . The wrapper script/tools/run_tests.py will process command line arguments and ensure
that the test suite is passed:

• a test output directory,
• a test run context,
• a list of test config ,
• a device inventory .

The real entry point is is for the test suite is inxv_leak_tools/test_execution/test_runner.py, which
receives the above objects from run_tests.py.

note that there ‘s an additional high level wrapper shell scriptrun_tests.sh which should be
used to run /tools/run_tests.py. This is just a helper script to ensure the suite can be
run in a platform agnostic way.

When the test suite run , it is does roughly does the following :

• discover all available test case ( class derive fromtestCase),
• iterating through each test configuration,
• create an instance of the test case class for the test,
• finds the device specified in the test config but looking through the device inventory,
• creates “connections” – roughly speaking, SSH connections – to all device,
• runs the tests, including:
  • setup and teardown,
  • handle success / failure ,
  • handling exceptions.

The test runner is tell will tell you what go wrong and summarise failure . It is ‘s ‘s similar to most unit
testing framework , but tailor to leak testing .

• learn about the current test case : TODO .
• Learn about building your own configurations: TODO.
• Learn about creating device inventories: TODO.