destructor performance, observables with large observer lists

[Peter Caspers-2]

Though probably of marginal importance, I spent some further time in
googling and experiments on the questions below.

I tried to stress the situation with the attached small test code. It has 3
parameters one can play with (the #define's) and two outputs.

"notFound" counts pointers that are put in a set and not found later on. If
this number is not zero there is a serious problem. However I did not manage
to provoke this on my environment, even with fat and/or many objects
allocated.

"fingerprint" counts the number of subsequently allocated pointers p and q
with not p < q. Different fingerprints (with same parameters) mean that code
relying on the ordering of pointers may show non deterministic or system
dependent behaviour. In fact I can produce different fingerprints with the
same executable on my system e.g. by starting it from a shell (2755), from
the IDE (3142) or double clicking in windows explorer (3193). Not very nice.

There might be real problems with the ordering of pointers on systems
without virtualized memory allocation. Like on dos or win3.x, but this is
not relevant any more for current systems, is it?

Finally the C99 standard actually does explicitly mention pointer
comparision in 6.5.8.

Peter


-----Ursprüngliche Nachricht-----
Von: Peter Caspers [mailto:[hidden email]]
Gesendet: Sonntag, 26. Juni 2011 20:28
An: 'Ferdinando Ametrano'
Cc: '[hidden email]'
Betreff: RE: [Quantlib-dev] destructor performance, observables with large
observer lists

Hi,

the main reason for the map<long,pointer> approach was to avoid using
comparison between pointers, which is to my understanding comparison of
memory addresses.

What if two objects pointed to do not stay in the same memory block for
example? Is the behaviour still well defined then? Even if it works
technically, program execution will probably depend on many things not
covered by the language specifications. It may be hard to get reproducible
results then?

Thank you and regards
Peter


-----Ursprüngliche Nachricht-----
Von: Ferdinando Ametrano [mailto:[hidden email]]
Gesendet: Sonntag, 26. Juni 2011 18:58
An: Plamen Neykov
Cc: [hidden email]; [hidden email]
Betreff: Re: [Quantlib-dev] destructor performance, observables with large
observer lists

On Sat, Jun 25, 2011 at 5:48 PM, Plamen Neykov
<[hidden email]> wrote:
> wouldn't it be simpler just to use std::set<Observer*> and save you the
trouble with the long id or am I missing something here?

great minds think alike... as a matter of fact in the trunk it has
already been switched to std:set on June 7th :-)

see
http://quantlib.svn.sourceforge.net/viewvc/quantlib?view=revision&revision=1
7788

I was more concerned with possible non-unique elements than
destructor's performance reason, and that's why I would keep it at
std::set instead of having it as template parameter.

Roland could you confirm that the trunk solution is OK for you?

BTW in the current trunk there is a MAJOR performance improvement if
you work in a real time environment with many changing rate quotes
between recalculations: I patched a bug which triggered many useless
notifications

thanks to all for the report and help: it's refreshing to have
contributors really stressing the library.

ciao -- Nando

----------------------------------------------------------------------------
--
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
QuantLib-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-dev

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
QuantLib-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-dev

[Plamen Neykov]

I guess, it is safe to say, that C++ (potentially with the exception of the MS
.NET managed C++), guarantees that the pointer to an allocated memory block
(from the viewpoint of the loaded executable image) will never change during
the current execution of that image.
Further, it is guaranteed that the pointers to different memory blocks will
always be different. In the context of std::set<Observer*>, this means that two
Observers will always have different (and immutable) pointers and it is not
possible to "lose" an Observer from the set.

However, the pointer to a block allocated after another block might compare
less than the pointer to the previously allocated block (this is an OS and
runtime dependent behaviour). In our case here, this will only have as a
consequence that the order of the notification of the registered observers can
not be determined - however, my impression was that the design of QuantLib
never required that the Observers are to be notified in the order of
registration.

(please note that I'm making here a distinction between the address and the
pointer to a memory block - it is possible to imagine that the address might
change, but a hypothetical C++ runtime might keep track of all pointers to
this block and change the pointers accordingly when the block's address
changes and as we are using Observer*, the runtime will have to change the
pointer to the Observer contained in the set and in the list so that your
"notFound" case cannot occur - however I'm not aware of the existence of any
such C++ runtime)

Cheers,
Plamen

On Saturday 02 Jul 2011 19:44:19 Peter Caspers wrote:

> Though probably of marginal importance, I spent some further time in
> googling and experiments on the questions below.
>
> I tried to stress the situation with the attached small test code. It has 3
> parameters one can play with (the #define's) and two outputs.
>
> "notFound" counts pointers that are put in a set and not found later on. If
> this number is not zero there is a serious problem. However I did not
> manage to provoke this on my environment, even with fat and/or many
> objects allocated.
>
> "fingerprint" counts the number of subsequently allocated pointers p and q
> with not p < q. Different fingerprints (with same parameters) mean that
> code relying on the ordering of pointers may show non deterministic or
> system dependent behaviour. In fact I can produce different fingerprints
> with the same executable on my system e.g. by starting it from a shell
> (2755), from the IDE (3142) or double clicking in windows explorer (3193).
> Not very nice.
>
> There might be real problems with the ordering of pointers on systems
> without virtualized memory allocation. Like on dos or win3.x, but this is
> not relevant any more for current systems, is it?
>
> Finally the C99 standard actually does explicitly mention pointer
> comparision in 6.5.8.
>
> Peter
>
>
> -----Ursprüngliche Nachricht-----
> Von: Peter Caspers [mailto:[hidden email]]
> Gesendet: Sonntag, 26. Juni 2011 20:28
> An: 'Ferdinando Ametrano'
> Cc: '[hidden email]'
> Betreff: RE: [Quantlib-dev] destructor performance, observables with large
> observer lists
>
> Hi,
>
> the main reason for the map<long,pointer> approach was to avoid using
> comparison between pointers, which is to my understanding comparison of
> memory addresses.
>
> What if two objects pointed to do not stay in the same memory block for
> example? Is the behaviour still well defined then? Even if it works
> technically, program execution will probably depend on many things not
> covered by the language specifications. It may be hard to get reproducible
> results then?
>
> Thank you and regards
> Peter
>
>
> -----Ursprüngliche Nachricht-----
> Von: Ferdinando Ametrano [mailto:[hidden email]]
> Gesendet: Sonntag, 26. Juni 2011 18:58
> An: Plamen Neykov
> Cc: [hidden email]; [hidden email]
> Betreff: Re: [Quantlib-dev] destructor performance, observables with large
> observer lists
>
> On Sat, Jun 25, 2011 at 5:48 PM, Plamen Neykov
>
> <[hidden email]> wrote:
> > wouldn't it be simpler just to use std::set<Observer*> and save you the
>
> trouble with the long id or am I missing something here?
>
> great minds think alike... as a matter of fact in the trunk it has
> already been switched to std:set on June 7th :-)
>
> see
> http://quantlib.svn.sourceforge.net/viewvc/quantlib?view=revision&revision=
> 1 7788
>
> I was more concerned with possible non-unique elements than
> destructor's performance reason, and that's why I would keep it at
> std::set instead of having it as template parameter.
>
> Roland could you confirm that the trunk solution is OK for you?
>
> BTW in the current trunk there is a MAJOR performance improvement if
> you work in a real time environment with many changing rate quotes
> between recalculations: I patched a bug which triggered many useless
> notifications
>
> thanks to all for the report and help: it's refreshing to have
> contributors really stressing the library.
>
> ciao -- Nando
>
> ---------------------------------------------------------------------------
> - --
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> QuantLib-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quantlib-dev

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
QuantLib-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-dev

[Peter Caspers-2]

Thanks for thinking about it, first of all.

>the design of QuantLib never required that the Observers are to be
>notified in the order of registration

I agree. Just wanted to point out that if a potential bug shows up or not
depending on this order, it might be hard to reproduce (even on the same
machine as the example shows) and to locate.

>Observers will always have different (and immutable) pointers and it is
>not possible to "lose" an Observer from the set
...
>a hypothetical C++ runtime might keep track of all pointers to
>this block and change the pointers accordingly when the block's address
>changes and as we are using Observer*, the runtime will have to change the
>pointer to the Observer contained in the set

In order to have O(log size) complexity for std::set.find() the ordering
among the pointers stored in the set must be used by the implementation
(some binary search) and therefore must be invariant during the whole
execution. In your hypothetical runtime it is therefore enough to have p<q
at t1 and q<p at t2 for two pointers p, q stored in the set to loose a
pointer, i.e. not being able to find it any more. Unless the runtime will
not only adjust the pointers in the set, but also triggers the set to
"re-sort" its elements, which sounds unlikely to happen. But as we are
talking about a hypothetical runtime, this is most probably really not
relevant...

Thanks a lot, Peter

-----Ursprüngliche Nachricht-----
Von: Plamen Neykov [mailto:[hidden email]]
Gesendet: Sonntag, 3. Juli 2011 03:21
An: [hidden email]
Cc: Peter Caspers
Betreff: Re: [Quantlib-dev] destructor performance, observables with large
observer lists

I guess, it is safe to say, that C++ (potentially with the exception of the
MS
.NET managed C++), guarantees that the pointer to an allocated memory block
(from the viewpoint of the loaded executable image) will never change during

the current execution of that image.
Further, it is guaranteed that the pointers to different memory blocks will
always be different. In the context of std::set<Observer*>, this means that
two
Observers will always have different (and immutable) pointers and it is not
possible to "lose" an Observer from the set.

However, the pointer to a block allocated after another block might compare
less than the pointer to the previously allocated block (this is an OS and
runtime dependent behaviour). In our case here, this will only have as a
consequence that the order of the notification of the registered observers
can
not be determined - however, my impression was that the design of QuantLib
never required that the Observers are to be notified in the order of
registration.

(please note that I'm making here a distinction between the address and the
pointer to a memory block - it is possible to imagine that the address might

change, but a hypothetical C++ runtime might keep track of all pointers to
this block and change the pointers accordingly when the block's address
changes and as we are using Observer*, the runtime will have to change the
pointer to the Observer contained in the set and in the list so that your
"notFound" case cannot occur - however I'm not aware of the existence of any

such C++ runtime)

Cheers,
Plamen

On Saturday 02 Jul 2011 19:44:19 Peter Caspers wrote:
> Though probably of marginal importance, I spent some further time in
> googling and experiments on the questions below.
>
> I tried to stress the situation with the attached small test code. It has
3
> parameters one can play with (the #define's) and two outputs.
>
> "notFound" counts pointers that are put in a set and not found later on.
If

> this number is not zero there is a serious problem. However I did not
> manage to provoke this on my environment, even with fat and/or many
> objects allocated.
>
> "fingerprint" counts the number of subsequently allocated pointers p and q
> with not p < q. Different fingerprints (with same parameters) mean that
> code relying on the ordering of pointers may show non deterministic or
> system dependent behaviour. In fact I can produce different fingerprints
> with the same executable on my system e.g. by starting it from a shell
> (2755), from the IDE (3142) or double clicking in windows explorer (3193).
> Not very nice.
>
> There might be real problems with the ordering of pointers on systems
> without virtualized memory allocation. Like on dos or win3.x, but this is
> not relevant any more for current systems, is it?
>
> Finally the C99 standard actually does explicitly mention pointer
> comparision in 6.5.8.
>
> Peter
>
>
> -----Ursprüngliche Nachricht-----
> Von: Peter Caspers [mailto:[hidden email]]
> Gesendet: Sonntag, 26. Juni 2011 20:28
> An: 'Ferdinando Ametrano'
> Cc: '[hidden email]'
> Betreff: RE: [Quantlib-dev] destructor performance, observables with large
> observer lists
>
> Hi,
>
> the main reason for the map<long,pointer> approach was to avoid using
> comparison between pointers, which is to my understanding comparison of
> memory addresses.
>
> What if two objects pointed to do not stay in the same memory block for
> example? Is the behaviour still well defined then? Even if it works
> technically, program execution will probably depend on many things not
> covered by the language specifications. It may be hard to get reproducible
> results then?
>
> Thank you and regards
> Peter
>
>
> -----Ursprüngliche Nachricht-----
> Von: Ferdinando Ametrano [mailto:[hidden email]]
> Gesendet: Sonntag, 26. Juni 2011 18:58
> An: Plamen Neykov
> Cc: [hidden email]; [hidden email]
> Betreff: Re: [Quantlib-dev] destructor performance, observables with large
> observer lists
>
> On Sat, Jun 25, 2011 at 5:48 PM, Plamen Neykov
>
> <[hidden email]> wrote:
> > wouldn't it be simpler just to use std::set<Observer*> and save you the
>
> trouble with the long id or am I missing something here?
>
> great minds think alike... as a matter of fact in the trunk it has
> already been switched to std:set on June 7th :-)
>
> see
>

http://quantlib.svn.sourceforge.net/viewvc/quantlib?view=revision&revision=

> 1 7788
>
> I was more concerned with possible non-unique elements than
> destructor's performance reason, and that's why I would keep it at
> std::set instead of having it as template parameter.
>
> Roland could you confirm that the trunk solution is OK for you?
>
> BTW in the current trunk there is a MAJOR performance improvement if
> you work in a real time environment with many changing rate quotes
> between recalculations: I patched a bug which triggered many useless
> notifications
>
> thanks to all for the report and help: it's refreshing to have
> contributors really stressing the library.
>
> ciao -- Nando
>
>

---------------------------------------------------------------------------

> - --
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> QuantLib-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quantlib-dev

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
QuantLib-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-dev