Benchmarks

These benchmarks compare the performance of RML versus the equivalent program written using the C++ Standard Library (“STL”).  All benchmarks are performed by the rm_bench.cpp program, which is included with the library.  The purpose of these tests is not to criticize the STL, or to compare particular compilers or libraries, but rather to reassure ourselves that RML is performing efficiently.

Test 1: One table, one index

This test looks at the basic table containing just a single index of integers.  This is basically what the STL containers are designed for, so the performance should be similar.

Containers:

RML:

RM_DEFINE_ROW_2(test1_row, unique<int>, x, int, y)

STL:

typedef std::map<int,int> test1_map;

Test 1a: Insert a row into the table.  Repeat for all rows.

Test 1b: Find a given row.  Repeat for all rows.

Test 1c: Sum a range of items.  Repeat for all rows.

Test 1d: Update the value of column 1.  Repeat for all rows.

Test 1e: Update the value of column 2.  Repeat for all rows.

Test 1f: Erase a given row.  Repeat for all rows.

Test 2: Single table, two indexes

This test involves a table with two indexes, one on each column.  This is implemented in the STL using two maps – one forwards and one backwards.

Containers:

RML:

RM_DEFINE_ROW_2(test2_row, unique<int>, x, unique<int>, y)

STL:

      std::map<int,int> from, to;

Test 2a: Insert a row into the table.  Repeat for all rows.

Test 2b: Find a given row.  Repeat for all rows.

Test 2c: Sum a range of items.  Repeat for all rows.

Test 2d: Update the value of column 1.  Repeat for all rows.

Test 2e: Update the value of column 2.  Repeat for all rows.

Test 2f: Erase a given row.  Repeat for all rows.

Test 3: One table, three indexes

This test involves adding a third index on a third column to Test 2.  This test highlights the efficiency of a multi-index implementation.

Containers:

RML:

RM_DEFINE_ROW_3(test3_row,

unique<int>, x,

unique<int>, y,

unique<int>, z)

STL:

      std::map<int,int> map1, map2, map3;

Test 3a: Insert a row into the table.  Repeat for all rows.

Test 3b: Find a given row.  Repeat for all rows.

Test 3c: Sum a range of items.  Repeat for all rows.

Test 3d: Update the value of column 1.  Repeat for all rows.

Test 3e: Update the value of column 2.  Repeat for all rows.

Test 3f: Erase a given row.  Repeat for all rows.

Test 4: Two tables, one index per table

This test has two tables, which are “joined”.  This tests the efficiency of the relational::select() implementation when iterating over two tables.  This is implemented using the STL with two maps.

Containers:

RML:

RM_DEFINE_ROW_2(test4_row1, unique<int>, x, unique<int>, y);

RM_DEFINE_ROW_2(test4_row2, unique<int>, x, unique<int>, y);

STL:

      std::map<int, int> map1, map2;

Test 4a: Insert a row into the table.  Repeat for all rows.

Test 4b: Find a given row.  Repeat for all rows.

Test 4c: Sum a range of items.  Repeat for all rows.

Test 4d: Erase a given row.  Repeat for all rows.

Test 5: Three tables, one index per table

This test has three tables, which are “joined”.  This tests the efficiency of the relational::select() implementation when iterating over three tables.  This is implemented using the STL with three maps.

Containers:

RML:

RM_DEFINE_ROW_2(test5_row1, unique<int>, x, int, y)

RM_DEFINE_ROW_2(test5_row2, unique<int>, x, int, y)

RM_DEFINE_ROW_2(test5_row3, unique<int>, x, int, y)

STL:

      std::map<int, int> map1, map2, map3;

Test 5a: Insert a row into the table.  Repeat for all rows.

Test 5b: Find a given row.  Repeat for all rows.

Test 5c: Sum a range of items.  Repeat for all rows.

Test 5d: Erase a given row.  Repeat for all rows.

Test 6: Filtering

This tests the efficiency of iterating over a table without an index.

Containers:

RML:

RM_DEFINE_ROW_3(test6_row,

unique<int>, id,

std::string, name,

int, num);

STL:

      typedef std::map<int, std::pair<std::string, int> > test6_map;

Test 6a: Find a single item in the table.  Repeat for all rows.

Test 6b: Sum a range of items matching a particular condition.  Repeat for all rows.

Results

Here are the results of the benchmarks. Results are given as times: lower time means better performance.

Hardware was a 1.7GHz Pentium M Centrino with 2MB cache and 512MB RAM.  The compiler was Microsoft C/C++ compiler 13.10.3052.

Conclusion

RML is efficient, matching or outperforming the C++ Standard Library in single-index tables, and easily outperforming it for multi-index tables.  Because of the use of templates, there are no run-time overheads associated with RML.  A solution based on run-time types (also known as variants) would perform much worse.

One cannot draw strong conclusions about these figures.  Factors such as how good the compiler is, the memory allocator and CPU caching will influence the numbers.  Because the C++ standard containers do not offer multi-index or reindex support, one would expect RML to perform much better in these cases because of less memory allocation.  A container that does provide multi-index and reindex support (e.g. Boost.MultiIndex) performs much better than the STL in these cases, and would in principle perform similarly to RML.

Performance is an important consideration when choosing a library.  In multi-user systems, high load servers, or portable devices, users will thank you for an efficient implementation.

Appendix: Output of rm_bench.exe 500000

Microsoft C/C++ 13.10.3052

                        insert  find    range   update1 update2 erase

1: std::map             563     141     437     2438    250     281

1: table<u>             219     62      438     328     140     172

   table<i>             219     125     422     437     250     203

2: std::map *2          1563    172     562     3313    3312    578

2: table<u,u>           313     62      438     344     328     203

   table<i,i>           312     125     438     437     438     234

3: std::map *3          1375    157     468     3375    3594    844

3: table<u,u,u>         422     62      453     360     281     219

4: std::map *2          1390    375     5282    0       0       578

4: table<u,u>*2         625     156     2062    0       0       375

5: std::map *3          1360    422     7765    0       0       828

5: table<u,u>*3         641     219     3859    0       0       500

6: STL                  31      6141    13125   0       0       0

6: RML                  16      5156    10516   0       0       0

GCC 4.0.1

                        insert  find    range   update1 update2 erase

1: std::map             375     141     406     1547    266     359

1: table<u>             359     125     454     640     235     250

   table<i>             343     235     437     781     469     328

2: std::map *2          735     156     437     1860    1859    719

2: table<u,u>           578     109     469     594     625     250

   table<i,i>           500     234     469     688     703     312

3: std::map *3          1125    141     422     1984    1984    985

3: table<u,u,u>         781     109     469     531     485     297

4: std::map *2          812     266     4156    0       0       703

4: table<u,u>*2         1156    250     3891    0       16      531

5: std::map *3          1187    391     7625    0       0       1031

5: table<u,u>*3         1063    406     7172    0       0       703

6: STL                  31      4657    9734    0       0       0

6: RML                  93      4657    10109   0       0       0