Skip to main content
Windward

Windward Performance Analysis

Engine Software Tested: 15.0.165.1
Date Tested: January 30th, 2017

Overview

Windward's engine is fast, very fast. The number one thing that we hear from our clients moving to Windward from competing solutions is the performance gain they receive from moving to our product. We realize that running performance tests under ideal conditions are not always reflected in real environments. We routinely update the performance PDF and this article with the latest versions of our engine benchmarks. However, we also provide this performance test as a part of our engine software package in both demo and production versions. We encourage you to test the Windward engine in your own native environment to see how it performs before you purchase.  

 

The steps below will outline how we test our product internally and how you can run these tests for yourself in your own environment. As always, we are curious to know your results and are always interested to see how our product performs in different environments. If you would like assistance running these tests and are willing to share your results then please feel free to contact us at support@windward.net and our Marketing team will reach out to you.

 

Download the Full Report here: Windward Performance Analysis Report - Windward-Engine-Performance.pdf

 

Additional reports for specific platforms and datasources are included at the end of this article.

What does this Article Cover?

 

The Numbers

The below tests all use the files in WindwardPerformance.zip.

  • The SQL Server contains the Northwind SQL database. The database server is located on the test machine.
  • The XML file is a 1.35M file which contains a dump of the SQL Server Northwind database. The file is on the local machine.
  • The template is 2 pages long and the report it generates is 17 pages long and is run 50 times.

The Measurement System

For this latest version of the Windward Engine, we tested the performance by creating a report 50 times and averaging the result. Performance results (number of pages created per second [PPS]) depended on which engine is used, what sort of data source is used, and the number of simultaneous threads called by the Windward application. When generating reports on a 4 cores system with two threads, our results ranged from 38 to 76 pages per second. Using 16 threads on the same 4 core system with the same reports boosted performance to 24 to 143 pages per second.

 

  • Processor: Intel® Xeon™ E5-2673 v3 @ 2.4GHz (1 physical socket; 4 virtual cores)
  • Memory: 14.0 GB
  • Operating system: Windows 10 Professional 64-bit
  • The Windward AutoTag template is a Microsoft® Word document two pages long. It contains text, images, headers, footers, multiple loops, and equations.
  • The generated report is a 17-page Word document (DOCX). The number of pages generated per second is not affected by the output format. Both DOCX and PDF reports were generated in the same amount of time. The report is run 50 times for each test.
  • Note that processor thrashing is occurring in the graphs below

 Datasource

Reports are generated using a DOCX template output to a DOCX report.

 

 

Windward Java Engine
Data Source Threads Pages per Second Peak Memory Usage
2

50.83

81 MB

4 86.49 85 MB
18 137.72 95 MB
16

142.79

109 MB

 

 

Windward .NET Engine
Data Source Threads Pages per Second Peak Memory Usage
2

73.91

94 MB

4 109.46 103 MB
18 53.23 115 MB
16

24.38

147 MB

 Datasource

 

Reports are generated using a DOCX template output to a DOCX report.

 

 

Windward Java Engine
Data Source Threads Pages per Second Peak Memory Usage
2

76.19

107 MB

4 110.58 134 MB
18 112.61 177 MB
16

115.00

252 MB

 

Windward .NET Engine
Data Source Threads Pages per Second Peak Memory Usage
2

38.15

178 MB

4 57.69 197 MB
18 59.19 250 MB
16

56.26

309 MB

Testing Notes

These Engine performance tests assume the worst case performance in several aspects, so your performance could easily be faster.

 

In this test:

  • We open the template each time the Engine generates a report. When you run the same template in bulk through the Windward Engine, it scans the template one time and caches it. This results in a speed increase as the time to read the template is removed from each successive run.
  • The data sources are loaded again each time, which may not be necessary in your use case.
  • Both DOCX and PDF reports were generated in the same amount of time.

A Note about processor thrashing

 

We ran these tests on a 4 core system.  Typically we recommend that you do not exceed 2 times the number of total cores on your target system.  You will notice in most of the tests above that performance starts to plateau after 8 threads (2 x 4 cores  = 8 threads).  This is a result of a phenomena called processor thrashing which is a state where the processor performs less productive work and more work swapping between all simultaneously running processes or threads.  This yields diminishing returns over time as the processor is spending more time trying to manage all simultaneous processes than actually processing them. 

 

As always, Windward recommends that you run our performance tests with difference thread values in order to find the sweet spot for your system.  The 2 times total system cores as threads is more of a guideline than a rule but we see this proven above in our standard testing.

How Do I Run the Tests?

The -performance:1234 option is available in the sample RunReport application shipped with both the .NET and Java engines. We strongly recommend that you run this program using your sample sets of templates and data on the server you will use.

 

One of the key factors to the performance of generating reports is based on the hardware being used. The greater the processing power and the larger memory your system has, will translate into improve performance of generating reports. That being said, here are some important stats to know in regards to generating reports.

 

There are a couple of generalizations we feel safe making about Windward Reports:

  • Its performance time is linear. It will take 10x longer to run 100,000 reports than it takes to run 10,000 reports.
  • For every CPU core you should generally have a maximum 2 threads generating reports.
  • Its memory usage is constant. Running 1,000 reports uses the same amount of memory as running 100,000 reports.
  • DOCX, XLSX, PPTX, PDF, HTML, and TXT reports all take about the same time.
  • File I/O is a big part of the time required.
  • Logic commands are the most expensive computationally.  IF/ELSE and SWITCH/CASE statements when used excessively will slow down your report generation (this is only noticeable if you have more than 150 tags of this type in a single template)

Performance Analysis Commands

Java Engine

1. Navigate to the directory where you installed the Java Engine

  • Windows: C:\Program Files (x86)\Windward Studios\Windward Java Engine\
  • Unix:  path where you extracted the Windward JAR files

 

2. Make sure all Windward JAR files are present in your CLASSPATH

  • Windows: this is done automatically, no further action needed
  • Unix: you will have to alter your CLASSPATH to include all JAR files in the Windward /lib directory

 

3. Change to the "test" directory

  • Windows: C:\Program Files (x86)\Windward Studios\Windward Java Engine\test
  • Unix: /Your_Windward_Path/test

 

4. Run the command below

 

Command

java net.windward.XmlReport.RunReport TEMPLATE_NAME OUTPUT_NAME DATA_CONNECTION –performance:NUM_REPORTS –threads:NUM_THREADS

.NET Engine

  1. Navigate to the directory where you installed the .NET Engine (C:\Program Files (x86)\Windward Studios\Windward .NET Engine\)

  2. Change to the "test" directory

  3. In this directory is the RunReport.exe program, a precompiled version of a command line implementation of the .NET Engine

  4. Run the command below

 

Command

RunReport.exe TEMPLATE_NAME OUTPUT_NAME DATA_CONNECTION –performance:NUM_REPORTS –threads:NUM_THREADS

Example

This example will run the template 10,000 times using 4 simultaneous threads to generate the reports in parallel.

RunReport.exe InternetMarketingReport.docx testreport.pdf -xml:INTMARKETING InternetMarketingData.xml -performance:10000 -threads:4

 

Run as a batch file

 

You can also save the following command in a .BAT file and then run it by passing in arguments for the performance (number of reports to run) and the threads (number of simultaneous reports to create)

 

ECHO *************************************************************************
ECHO * Usage:  run_performance.bat NUM_REPORTS NUM_THREADS *
ECHO *************************************************************************
RunReport.exe InternetMarketingReport.docx testreport.pdf -xml:INTMARKETING InternetMarketingData.xml -performance:%1 -threads:%2

RunReport.exe command Example

When executing the RunReport program from the command line of the Windward Engine with no parameters you will see the following output below will all options that can be used. The -performance and -threads parameters are bolded below.

 

usage: RunReport template_file output_file [-xml xml_file | -sql connection_string | -oracle connection_string | -ole oledb_connection_string | -playback[:name] recorder_file] [key=value | ...]
       
       -performance:123 - will run the report 123 times, output file is used for directory and extension for reports
       -threads:4 - will create 4 threads when running -performance.

 

The full RunReport Reference Article can be found here.

Platform and Datasource Specific Test Results Documents

Java Engine Performance Test Results

.NET Engine Performance Test Results