destructor performance, observables with large observer lists
Locked
destructor performance, observables with large observer lists
 
|
Hi all,
we would like to share an observation here that we have made in a project developing an application based on QuantLib. The application loads market data and relatively large trade portfolio from a database, builds term structures, engines and instrument objects and finally does some pricing and portfolio analytics. Term structures and market data are stored "globally" (in some singleton manager) with many instruments "observing" relatively few term structures. We kept term structures "global" to allow for scenario analysis (such as zero rate shifts etc.) across the portfolio. What we have noticed is that the application spends significant amount of time at the very end (just before main returns 0). For example, the program loads/constructs about 4000 vanilla deals and related term structures in a minute, prices the deals within seconds and then spends more than 2 minutes terminating itself. Somewhat unsettled by this we investigated the delay further. Logging messages from all kinds of destructors showed that the time is spend - maybe not surprisingly - on instrument destruction. Moreover, the destruction time seemed to grow non-linearly with portfolio size (only 2sec destruction time for about 700 vanilla deals). These times were taken on a MacBook Pro with Intel Core 2 Duo, 2.8 GHz, gcc 4.2, optimization -O2. The attached test program observer.cpp demonstrates this behavior. If you run it with "./observer 2 2000 1" it builds and prices 2000 identical copies of vanilla swaps all linked to the same global term structure. The origin of this termination delay is in the observer/observable chains, see ql/pattern/observable.hpp: A single term structure is "observed" in the example by a large number of instruments. When an instrument is destroyed, it first unregisters from its observables (one term structure in our case and several floating rate coupons). This involves a lookup of the given observer in the observable's long list of observers (the term structure has thousands of observers). This lookup is the time consuming step. If we comment out all register/unregister code in observable.hpp, the delay vanishes completely. We have come up with the attached slight enhancement of observable.hpp where we have basically replaced list<Observer*> by map<long,Observer*> with some auto-generated id of type long so that the elements can be compared and retrieved more efficiently. With this change the delay upon program termination is gone, while we don't see noticeable increase in time spent on registering observers. We have also checked that the QuantLib test suite (1.0.1) completes without errors with both original and new observable.hpp. Did anybody else make a similar observation? Do you think it is worth adding the amendment or something similar to QuantLib? We would be interested in any feedback. Kind regards, Roland ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 _______________________________________________ QuantLib-dev mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/quantlib-dev |
observer.cpp (11K)
Re: destructor performance, observables with large observer lists
|
|
Re: destructor performance, observables with large observer lists
|
|
Re: destructor performance, observables with large observer lists
|
|
In reply to this post by Plamen Neykov
Hi Plamen,
you are right, using std::set instead of std::list does it as well, same speed of termination as the previous proposal, and the test suite shows no errors. I'd definitely prefer this enhancement, see attached. Does anybody see a reason/need to keep track of the order of observer registration that we got with the original observable.hpp? Regards, Roland On 25.06.2011, at 17:48, Plamen Neykov wrote: > A very quick comment - wouldn't it be simpler just to use std::set<Observer*> and save you the trouble with the long id or am I missing something here? > > Regards, > Plamen > > On 25 Jun 2011, at 14:37, Roland Lichters <[hidden email]> wrote: > >> Hi all, >> >> we would like to share an observation here that we have made in a project developing an application based on QuantLib. The application loads market data and relatively large trade portfolio from a database, builds term structures, engines and instrument objects and finally does some pricing and portfolio analytics. Term structures and market data are stored "globally" (in some singleton manager) with many instruments "observing" relatively few term structures. We kept term structures "global" to allow for scenario analysis (such as zero rate shifts etc.) across the portfolio. >> >> What we have noticed is that the application spends significant amount of time at the very end (just before main returns 0). For example, the program loads/constructs about 4000 vanilla deals and related term structures in a minute, prices the deals within seconds and then spends more than 2 minutes terminating itself. Somewhat unsettled by this we investigated the delay further. Logging messages from all kinds of destructors showed that the time is spend - maybe not surprisingly - on instrument destruction. Moreover, the destruction time seemed to grow non-linearly with portfolio size (only 2sec destruction time for about 700 vanilla deals). These times were taken on a MacBook Pro with Intel Core 2 Duo, 2.8 GHz, gcc 4.2, optimization -O2. The attached test program observer.cpp demonstrates this behavior. If you run it with "./observer 2 2000 1" it builds and prices 2000 identical copies of vanilla swaps all linked to the same global term structure. >> >> The origin of this termination delay is in the observer/observable chains, see ql/pattern/observable.hpp: A single term structure is "observed" in the example by a large number of instruments. When an instrument is destroyed, it first unregisters from its observables (one term structure in our case and several floating rate coupons). This involves a lookup of the given observer in the observable's long list of observers (the term structure has thousands of observers). This lookup is the time consuming step. If we comment out all register/unregister code in observable.hpp, the delay vanishes completely. >> >> We have come up with the attached slight enhancement of observable.hpp where we have basically replaced list<Observer*> by map<long,Observer*> with some auto-generated id of type long so that the elements can be compared and retrieved more efficiently. With this change the delay upon program termination is gone, while we don't see noticeable increase in time spent on registering observers. >> >> We have also checked that the QuantLib test suite (1.0.1) completes without errors with both original and new observable.hpp. >> >> Did anybody else make a similar observation? Do you think it is worth adding the amendment or something similar to QuantLib? >> We would be interested in any feedback. >> >> Kind regards, >> Roland >> >> >> <observer.cpp> >> >> >> <observable.hpp> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> All the data continuously generated in your IT infrastructure contains a >> definitive record of customers, application performance, security >> threats, fraudulent activity and more. Splunk takes this data and makes >> sense of it. Business sense. IT sense. Common sense.. >> http://p.sf.net/sfu/splunk-d2d-c1 >> _______________________________________________ >> QuantLib-dev mailing list >> [hidden email] >> https://lists.sourceforge.net/lists/listinfo/quantlib-dev ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ QuantLib-dev mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/quantlib-dev |
Re: destructor performance, observables with large observer lists
|
|
Re: destructor performance, observables with large observer lists
|
|
Re: destructor performance, observables with large observer lists
|
|
| Free forum by Nabble | Edit this page |