Testing with Microcks

Introduction

It is likely you experienced the painful situation of deploying to production only to find out that an API service you integrate with has broken the contract. How can we effectively ensure this does not happen?

Microcks offers mocks but can also be used for Contract conformance testing of API or services being under development. You spend a lot of time describing request/response pairs and matching rules: it would be a shame not to use this sample as test cases once the development is on its way!

Conformance testing

You find on the internet many different representations of how the different testing techniques relates to one another and should be ideally combine into a robust testing pipeline. At Microcks, we particularly like the Watirmelon representation below. Microcks clearly allows you to realize Automated API Tests and focus more precisely on Contract or Conformance testing.

The purpose of Microcks tests is precisely to check that the Interaction Contract - as represented by an OpenAPI or AsyncAPI specification, a Postman collection or whatever supported artifact - consumer and producer agreed upon is actually respected by the API provider. In other words: to check that an implementation of the API is conformant to its contract.

In order to help you getting confidence into your implementations, we developed the Conformance index and Conformance score metrics that you can see on the top right of each API | Service details page:

This metrics are available from the 1.6.0 version of Microcks.

Conformance metrics

The Conformance index is a kind of grade that estimates how your API contract is actually covered by the samples you’ve attached to it. We compute this index based on the number of samples you’ve got on each operation, the complexity of dispatching rules of these operation and so on… It represents the maximum possible conformance score you may achieve if all your tests are successfull.

The Conformance score is the current score that has been computed during your last test execution. We also added a trend computation if things are going better or worse comparing to your history of tests on this API.

Once you have activated labels filtering on your repository and have ran a few tests, Microcks is also able to give you an aggregated view of your API patrimony in termes of Conformance Risks. The tree map below is displayed on the Dashboard page and represents risks in terms of average score per group of APIs (depending on the concept you chose it could be per domain, per application, per team, …)

This visualization allows you to have a clear understanding of your conformance risks at first glance!

Running tests

From the page displaying basic information on your API or Service mocks, you have the ability to launch new tests against different endpoints that may be representing different environment into your development process. Hitting the NEW TEST… button, leads you to the following form where you will be able to specify a target URL for the test, as well as a Runner—a testing strategy for your new launch:

While it is convenient to launch test on demand manually, it may be interesting to consider launching new tests automatically when a new deployment of the application occurs for example… Microcks allows you such automation by offering API for ease of integration. See here for more details).

Test parameters

Service under test

Service under test is simply the reference of the API/Service specification we’d like to test. This a couple of Service Name and Service Version.

Test Endpoint

The Test Endpoint is simply a URI where a deployed component is providing API specification endpoint. In the testing literature, this is usually defined as the URI of the System Under Test.

HTTP based APIs

For HTTP based APIs (REST, SOAP, GraphQL or gRPC), this is a simple URL that should respect following pattern:

http[s]://{service.endpoint.url:port}/{service.path}

Event based APIs

For Event based API through Async API testing, pattern is depending on the protocole binding you’d like to test.

Kafka

Kafka Test Endpoint have the following form with optional parameters placed just after a ? and separated using & character:

kafka://{kafka.broker.url:port}/{kafka.topic.name}[?param1=value1&param2=value2]

As an example, you may have this kind of Test Endpoint value: kafka://mybroker.example.com:443/test-topic?registryUrl=https://schema-registry.example.com&registryUsername=fred:letmein&registryAuthCredSource=USER_INFO

MQTT

MQTT Test Endpoint have the following form with no optional parameters:

mqtt://{mqtt.broker.url:port}/{mqtt.topic.name}

AMQP

AMQP 0.9.1 Test Endpoint have the following form with optional parameters placed just after a ? and separated using & character:

amqp://{amqp.broker.url:port}/[{amqp.vhost}/]{amqp.destination.type}/{amqp.destination.name}[?param1=value1&param2=value2]

amqp.destination.type is used to specify if we shoulf connect to either a queue (use the q value) or an exchange speciyfing its type: d dor direct, f for fanout, t for topic, h for headers. Then you have to specify either the queue or exchange name in amqp.detaintion.name.

Depending on the type of destination, you will need additional optional parameters as specified below:

As an example, you may have this kind of Test Endpoint values: amqp://rabbitmq.example.com:5672/h/my-exchange-headers?h.h1=h1&h.h2=h2 or amqp://rabbitmq.example.com:5672/my-vhost/t/my-exchange-topic?routingKey=foo

WebSocket

WebSocket Test Endpoint have the following form with no optional parameters:

ws://{ws.endpoint.url:port}/{channel.name}

NATS

NATS Test Endpoint have the following form with no optional parameters:

nats://{nats.endpoint.url:port}/{queue-or-subject.name}

Test Runner

As stated above, Microcks offers different strategies for running tests on endpoints where our microservice being developed are deployed. Such strategies are implemented as Test Runners. Here are the default Test Runners available within Microcks:

Timeout

Depending on the type of Service or Tests you are running, the specification of a Timeout maybe mandatory. This is a numerical value expressed in milliseconds.

Secret

Depending on the Test Endpoint you are connecting to, you may need additional authentication information - like credentials or custom X509 Certificates. You may reuse External Secrets that has been made available in the Microcks installation by the administrator.

Getting tests history and details

Tests history for an API/Service is easily accessible from the API | Service summary page. Microcks keep history of all the launched tests on an API/Service version. Success and failures are kept in database with unique identifier and test number to allow you to compare cases of success and failures.

Specific test details can be visualized : Microcks also records the request and response pairs exchanged with the tested endpoint so that you’ll be able to access payload content as well as header. Failures are tracked and violated assertions messages displayed as shown in the screenshot below :