Selecting a tool for workload simulation can depend on a number of things, including, but not limited to, the features it provides, the system to be tested and the reason for using it – is it a temporary deployment for ad-hoc troubleshooting, or something more long term that is part of a performance engineering strategy? In the case of a strategic framework the implementation should also align with the organisation / team’s technology preferences and way of working, as well as the environment it is to be used in (e.g. cloud, on premise). Additional considerations may include but are not limited to:
- Tests: how are they created, are they programmable, what is the engine ?
- Scenarios: what features are provided for creating and controlling workloads ?
- Protocols: does it support the application protocols in use ?
- Extensibility: can it be extended to cover new protocols or interfaces later down the line ?
- Monitoring: does it integrate with common monitoring platforms (e.g. InfluxDB, Elastic) ?
- Data: what data management features are available (e.g. input formats, real-time data sharing) ?
- Automation: how well does it integrate with CI/CD tools, does it support threshold criteria ?
- Scalability: how is this best achieved, what does the tool provide for coordination ?
- Infrastructure: what type and how much is required to generate the target workloads ?
- Licensing: commercial vs open source, are any open source features paid for ?
- Management: how can results and collateral be managed ?
- Reporting: options available, can raw data be accessed, what is the precision of response times ?
- Development: how active is the community, when was it last updated ?
If feasible it can be a useful exercise to shortlist a set of tools and evaluate them against a benchmark environment to see first hand how each is used to design a test, what the quality of reporting is like, and also estimate how much capacity may be required to simulate the test workloads.
A short list of open source tools as of November 2022 can be found here.