The quickest way to get a feel for the Query Engine is to use the compose file.

For consistency this document assumes you are using Linux or WSL, very similar commands should work on Windows.

Download that file and save it to a new directory. Open a shell and CD to that directory, then run:

docker compose -f query-engine-compose.yml -p query-engine up -d

Then navigate to http://localhost:2000/ in your favourite browser.

Details

This compose file needs to be run on a single container host (i.e. not Swarm or Kubernetes) with localhost ports 2000-2003 free (as well as the ports required by Jaeger: 5775/udp, 6831/udp, 6832/udp, 5778, 4317, 4318, 16686, 14268). It spawns five containers: MySQL, Microsoft SQL Server, PostgreSQL, Jaeger all-in-one and Query Engine in Design Mode.

The Jaeger container can be removed without making any further changes - no tracing data will be captured, but the Query Engine will otherwise work.

The configuration of the Query Engine container uses the sampleDataLoads configuration to set up a database on each of these DBMS platforms with a very similar structure and data. This data is all set up in transient storage - when the containers are deleted the sample data will go with them (though the containers can be restarted without losing it).

The Microsoft SQL Server and PostgreSQL containers can be removed from the compose file, but you should also remove their configured sampleDataLoads from the QueryEngine environment. The sample pipelines that use these containers won’t work.

The MySQL container can also be removed, but that is also used by Query Engine for its own persistence. In addition to removing the sampeDataLoads configuration for MySQL you will need to either change the audit.persistence to another destination, or remove it completely to use in-memory audit and login state.

Whenever the Query Engine starts it tries to access (and monitor) its baseConfigPath (by default this is "/var/query-engine"; it can be changed if necessary, though you should prefer using volume mounts to point this location to your files rather than reconfiguring it). In production use the baseConfigPath is where you will store the query files that the Query Engine uses. Whenever the Query Engine boots, if the baseConfigPath does not exist it will attempt to create a directory (and parent directories) there. If the baseConfigPath does exist and isn’t a directory the Query Engine will fail to start. If the baseConfigPath is an empty directory the Query Engine will copy the sample queries to that location - the only way to disable this behaviour is to create a file in the directory before starting the Query Engine.

The sample compose file explicitly specifies the baseConfigPath for clarity, but does not actually change it from the default value. The directory "./pipelines" is mounted to /var/query-engine. For real use this volume should point to a git (or other SCM) repository for your pipeline definitions.

Putting this behaviour together means that when the Query Engine starts using the provided compose file it is a demonstrable system that can be experienced with minimal additional work.

Note
 Although the system deployed is fully operational the History section of the UI will not return any data because you have not logged in. The data is recorded and you are encouraged to have a look at the mysql database (localhost:2001/test, credentials are in the compose file) to see it.

Introduction

Configuration