Financial Software Performance Benchmarking

[Jace A Mogill]

Hello,

AMD is looking for financial software benchmarks to
showcase both new hardware and business applications.

QuantLib is very promising due to it's wide use and
diverse feature set.  However, the benchmark that comes
with it runs 27 different experiments in less than 90
seconds, and only requires about 20MB of memory.  In an
ideal benchmark, each experiment would take at least
one minute and would use several GB of memory.  Even
larger experiments are welcome as they allow the
benchmark to chart performance improvements over the
years without becoming "too simple".

If you have a non-proprietary financial simulation
benchmark (does not have to be based on QuantLib),
that run for minutes or hours, or have large memory
requirements, AMD would like work with you on
developing the benchmark and publishing results.
Thanks,


                     -J


==================================================================
Jace A Mogill                             AMD Austin
Senior Member of Technical Staff          7171 Southwest Parkway
Performance CoE, Central Engineering      MS: 400.1B
(+1) 512.602.4916                         Austin, TX  78735


------------------------------------------------------------------------------
Register Now & Save for Velocity, the Web Performance & Operations
Conference from O'Reilly Media. Velocity features a full day of
expert-led, hands-on workshops and two days of sessions from industry
leaders in dedicated Performance & Operations tracks. Use code vel09scf
and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

[Bojan Nikolic]

Jace A Mogill <[hidden email]> writes:

> QuantLib is very promising due to it's wide use and
> diverse feature set.  However, the benchmark that comes
> with it runs 27 different experiments in less than 90
> seconds, and only requires about 20MB of memory.  In an
> ideal benchmark, each experiment would take at least
> one minute and would use several GB of memory.  

If fact, it is not a bad approximation to real life to simply loop
some of the tests in the quantlib benchmark many times. The reason is
that in practice one often needs to value a large portfolio (say 1000)
of instruments for a wide range of different scenarios (one can easily
get to 10000) and computationally this is similar to simply repeating
the valuations. Further some valuation strategies are actually based
on running many multiple scenarios of simpler models.

Best,
Bojan


--
Bojan Nikolic          ||          http://www.bnikolic.co.uk

------------------------------------------------------------------------------
Register Now & Save for Velocity, the Web Performance & Operations
Conference from O'Reilly Media. Velocity features a full day of
expert-led, hands-on workshops and two days of sessions from industry
leaders in dedicated Performance & Operations tracks. Use code vel09scf
and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

[Dirk Eddelbuettel]

On 30 April 2009 at 15:28, Jace A Mogill wrote:
| AMD is looking for financial software benchmarks to
| showcase both new hardware and business applications.
|
| QuantLib is very promising due to it's wide use and
| diverse feature set.  However, the benchmark that comes
| with it runs 27 different experiments in less than 90
| seconds, and only requires about 20MB of memory.  In an

Did you try the quantlib-test-suite?  Someone had noticed recently that the
quantlib-test-suite (which we ship in Debian as part of the libquantlib0-dev
package) bombs out with a seg fault (which has since been fixed; had to do
with starting that subtest on a non-trading day).  That ran for 18 minutes of
my fairly recent, fairly fast dual-core.  

And as you get the source, you can always parameterize and increase the 'big
N' number for Monte Carlo runs by a scale parameter.

Hth, Dirk

| ideal benchmark, each experiment would take at least
| one minute and would use several GB of memory.  Even
| larger experiments are welcome as they allow the
| benchmark to chart performance improvements over the
| years without becoming "too simple".

--
Three out of two people have difficulties with fractions.

------------------------------------------------------------------------------
Register Now & Save for Velocity, the Web Performance & Operations
Conference from O'Reilly Media. Velocity features a full day of
expert-led, hands-on workshops and two days of sessions from industry
leaders in dedicated Performance & Operations tracks. Use code vel09scf
and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

[MH_quant]

Hello,


-----Original Message-----
Even
larger experiments are welcome as they allow the
benchmark to chart performance improvements over the
years without becoming "too simple".

If you have a non-proprietary financial simulation
benchmark (does not have to be based on QuantLib),
that run for minutes or hours, or have large memory
requirements, AMD would like work with you on
developing the benchmark and publishing results.
Thanks,
-----Original Message-----



Well, I am doing Vega stress tests with quantlib on a local vol surface for
an asian path-dependent option. I need Monte Carlo Simulations and lots of
pathes (which means high dimensional random variable matrices). It keeps my
dual core busy for over 50 hours.

If your AMD can do it considerably faster, I am willing to buy it :)


Greetings
Michael


------------------------------------------------------------------------------
Register Now & Save for Velocity, the Web Performance & Operations
Conference from O'Reilly Media. Velocity features a full day of
expert-led, hands-on workshops and two days of sessions from industry
leaders in dedicated Performance & Operations tracks. Use code vel09scf
and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

[Klaus Spanderen-2]

Hi Jace,

please find attached a first draft of a modified version of the QuantLib
benchmark. This version now supports the following command line arguments

 --large  : runs each benchmark multiple times (20). In the future this could
also enable "larger" (in term of memory usage) versions of some benchmarks.

 --nprocs=<#processes>: starts a master process waiting for #processes-1
childs to join to run the benchmarks in parallel.

 --child : starts a child process.

The master reports the performance of the sum of all processes. E.g. on a Unix
machine

./quantlib-benchmark --large --nprocs=2 & ./quantlib-benchmark --large --child

starts a "large" benchmark with two processes (running the benchmark suite
twice).

This version uses boost::interprocess to synchronize the processes and to
calculate the overall performance. Therefore you need to have at least
boost-1-35 installed. (I haven't had the time to check the program on a
Windows box but it should work as well.)
 
regards
 Klaus


------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

[Klaus Spanderen-2]

upps..sorry, wrong file version attached, this one will work with nprocs > 2

regards
 Klaus



------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

[Luigi Ballabio]

On Thu, 2009-04-30 at 15:28 -0500, Jace A Mogill wrote:
> AMD is looking for financial software benchmarks to
> showcase both new hardware and business applications.

Jace,
        I see Klaus already sent you a benchmark.  Please keep us informed; it
seems like an interesting project, and I guess there are quite a few
people on this list that would like to know the results.

Luigi


--

Ogden's Law:
The sooner you fall behind, the more time you have to catch up.



------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

[Jace A Mogill]

Hi Luigi et al,

Thanks to everyone who expressed interest.  I
implemented Bojan Nikolic's suggestion to add an outer
loop around the existing benchmarks to approximate the
simulation of large portfolio.  This scaled
the execution time of the benchmark linearly, so I'm
able to run the length of time of my choosing, but the
memory use is unchanged.

One of the purposes of the benchmark is to showcase all
the cores on the system, so I was glad to receive
Klaus' parallel version.  As you might imagine, some of
the newer systems have lots of cores -- by year's end
our processors will have 6 cores, meaning 4-socket
systems will have 24 cores.  Our experience is that
communication via shared files does not scale
efficiently to that many tasks, so I did not try to get
that to scale.

I tried two different approaches to parallelizing the
benchmark:

  - Parallelizing the outer loop I introduced using
    OpenMP.  Unfortunately there were problems with
    memory allocation in QuantLib or BOOST or both, and
    the program would fail with a segfault.

  - Shell script to fork off one job per core.  Because
    there is no communication between tasks, this is
    essentially a hybrid of Klaus' and Bojan's
    approaches.  This approach is scaling linearly with
    the number of cores used because it is not very
    memory intensive.

If anyone has experience with using QuantLib or BOOST
in a shared memory parallel context (ie: OpenMP or
Pthreads), or already knows there are single-thread
dependencies in the code which prohibit this type of
parallelism, I would appreciate any tips or hints!

I am still looking at other optimizations, so I'm still
collection information and am not ready to report any
results yet.  Thanks again for your interest,


                        -J


==================================================================
Jace A Mogill                             AMD Austin
Senior Member of Technical Staff          7171 Southwest Parkway
Performance CoE, Central Engineering      MS: 400.1B
(+1) 512.602.4916                         Austin, TX  78735



On Friday, May 8, 2009 at 15:59:17, Luigi Ballabio <[hidden email]> wrote:

> On Thu, 2009-04-30 at 15:28 -0500, Jace A Mogill wrote:
> > AMD is looking for financial software benchmarks to
> > showcase both new hardware and business applications.
>
> Jace,
> I see Klaus already sent you a benchmark.  Please keep us informed; it
> seems like an interesting project, and I guess there are quite a few
> people on this list that would like to know the results.
>
> Luigi
>
>
> --
>
> Ogden's Law:
> The sooner you fall behind, the more time you have to catch up.
>
>
>

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users