Re: [Quantlib-dev] Linux Port

[Ferdinando M. Ametrano-2]

>         I sent nando an initial embryonicGNU/Linux port. Generates makefiles
>dynamically, almost everything compiles. More when I have another five
>minutes.
thank you Bernd

>Initial impressions:
>
>         I am unhappy about the heavy use of templates and the stl. Its
> going to
>bite you on many platforms (guys there is a whole world out there apart from
>Macs and Windows machines...) and the bloat is going to be considerable.
Luigi will reply to this one. In the meantime can you elaborate on which
platform you think we will have problems?
QuantLib compiles on Win32, Mac and (hopefully soon) on Linux.
Sun Solaris is the only other platform used in financial world I can think of.

>         Is the license compatible with the GPL?
Nice to hear such a question from a KDE developer ;-)
Yes it is. The XFree-86 style licence (also called BSD style licence)
classifies QuantLib as both Free Software and Open Source, as you can see
following the links in http://quantlib.sourceforge.net/license.html

>  Has someone verified with Richard?
I had email exchanges with Richard Stallman before announcing QuantLib. He
advocated GPL or LGPL of course.
I chose XFree-86 style licence because I want software companies and/or
researchers to be able to release proprietary model based on QuantLib under
more restrictive copyright. I think this is a requirement if we want
QuantLib to become a widespread adopted standard, since proprietary model
are a competitive advantage in the trading rooms.
This is also a requirement from my company, even if I hope and think that
we will GPL our products.
I will get back to Richard to let him known about the QuantLib project, and
to get a confirmation that my wording of XFree-86 is good enough to be GPL
compatible.

Of course my approach has some drawback.
We cannot use GPLed libraries to preserve the QuantLib licence, since GPL
would contaminate QuantLib. We can use LGPL libraries.

>             We need to talk seriously about the architecture sometime soon.
I posted a to do list as a starter
(http://lists.sourceforge.net/archives//quantlib-users/2000-December/000004.html)
This is not talking about architecture yet, but it is a starting point.
I also owe an answer to Gilbert Peffer message
(http://lists.sourceforge.net/archives//quantlib-users/2000-December/000005.html)

Feel free to start a design thread on this mailing list if you have some
proposal

ciao -- Nando

[Ferdinando Ametrano-6]

> > We cannot use GPLed libraries to preserve the QuantLib licence, since GPL
> > would contaminate QuantLib. We can use LGPL libraries.
>
>Well, lets see whether we can survive without  great code such as the stats
>libs of R etc.
I have a pragmatic approach, it's matter of evaluating what will help more:
GPL code or support from GPL-adverse financial software/consulting companies.

ciao -- Nando

[Luigi Ballabio-3]

At 07:18 PM 12/18/00 +0100, Ferdinando Ametrano wrote:
>>Initial impressions:
>>
>>         I am unhappy about the heavy use of templates and the stl. Its
>> going to bite you on many platforms (guys there is a whole world out
>> there apart from Macs and Windows machines...) and the bloat is going to
>> be considerable.
>Luigi will reply to this one.

Here I am. Of course you're right about compiler support (we already had
quite a few headaches because of this) and about the bloat. I am also aware
that, as Peter Schmitteckert wrote,

>for a library, I prefer virtual classes/functions to achieve conveniency
>and flexibility. You can extend classes with virtual function, withouot
>recompiling or changing the interface. With templates you can't (normally) not.
>...and remember, virtual fuctions are just a pointer-derefencation, and
>that's cheap on microprocessors.

Well, I'll try and explain the reasoning behind the current template
implementation, of which-I think-the finite difference package is the most
blatant example. (P.S. Sorry - it turned out to be a long e-mail, but I
wanted to put all my cards on the table so that we can continue discussing
the issue with the complete picture in mind)

First and foremost, the virtual classes approach is one that we definitely
want to pursue - expecially as we are exporting the library towards Python
(and hopefully other languages) where we cannot exploit the C++ template
mechanism.
In short, I surely want to be able to use dynamic polymorphism to switch
between FiniteDifferenceModel<CrankNicolson<TridiagonalOperator> >,
FiniteDifferenceModel<BackwardEuler<MultigridOperator> >,
FiniteDifferenceModel<MyVerySpecificPadeApproximation>,
FiniteDifferenceModel<WhateverYouLike>, and even TreeModel<WhateverItNeeds>.
This can be (more or less) easily accomplished by inheriting template<class
Evolver> FiniteDifferenceModel<Evolver> from some more generic interface
class. However, this is a part of the global design that didn't make into
the library yet. We just uploaded the inner part of it.

At the same time, while very flexible with respect to usability by the
final user, dynamic polymorphism is somehow less flexible with respect to
developing new classes - because it forces such classes to belong to a
particular hierarchy. I see two main disadvantages of this approach:
1) multiple inheritance rears its ugly head, which I would avoid if
possible. This is very likely to be a only a prejudice of mine - so feel
free to ignore it - but it is getting much too close to virtual base
classes for my liking.
2) if you decide that, e.g., some sparse array class you found in another
library would be just right for your particular problem, you'll have to either:
   a) modify that class so that it inherits from the base class for arrays
in our brand new polymorhpic framework, which is not always possible, or
   b) write a wrapper class which contains the sparse array and inherit
from the base one.
In a template framework, you can just go ahead and write your differential
operator using the sparse array class as it is, and its full compatibility
with the framework is just a trait specialization away.

Furthermore, if different array classes - possibly including sparse ones -
or different operator classes  - tridiagonal, band diagonal, sparse, you
name it - are to be derived from base classes, then it is very likely that
we will have to declare as virtual even very basic functionalities (i.e.,
operator[]) which tend to get in the innermost loops of the code and get
called humongous number of times. Now, while it is true that virtual calls
are cheap nowadays, it is also true that precious few optimizers inline
them. A simple algorithm for multiplying a matrix and a vector would either:
a) be written in a generic way, in which case it would jump in and out the
respective operator[] for most of the time, which will add up to quite a
loss in efficiency, or
b) be specialized for the concrete operators and array - however, this
would need double dispatching, which is not easier to write and maintain
than templates.

Finally, there's still one thing that static polymorphism can do and
dynamic polymorphism can't, and that's generic algorithms on iterators. You
simply can't derive all iterators from a single class as Java does, because
you can't add a parent class to C pointers. But I don't think you need to
hear this from me :)

Well, I'm open to suggestions. In short, my point was: I'd like to be able
to use templates in the inner parts of the models - arrays, operators... -
for the kind of flexibility I explained above. I'd also like to use dynamic
polymorphism in the outer parts - the complete models and their interface -
for the kind of flexibility users, including myself, want.
Once the design is refined, we could have the best of both worlds - and of
course a bit of the worst of both, compiler support included :) But it is a
trade-off that I would personally accept.

Thanks for listening (man, was this long!)

                 Luigi

[Gilbert Peffer]

Some headaches I had using templates:
A few months ago, I wrote some C++ template code using STL templates (which
in turn contained nested templates). I had problems looking at the value of
the variables in my Visual C++ debugger because some of the templates were
resolved into over 20 lines or so. Perhaps I didn't find the option that
would allow me to collapse the expanded template name into something more
managable. If there isn't, then templates can make your debugging life much
more difficult.
Regards
Gilbert

> -----Mensaje original-----
> De: [hidden email]
> [mailto:[hidden email]]En nombre de Luigi
> Ballabio
> Enviado el: 19 December 2000 16:51
> Para: [hidden email]
> Asunto: [Quantlib-users] Templates
>
>
> At 07:18 PM 12/18/00 +0100, Ferdinando Ametrano wrote:
> >>Initial impressions:
> >>
> >>         I am unhappy about the heavy use of templates and the stl. Its
> >> going to bite you on many platforms (guys there is a whole world out
> >> there apart from Macs and Windows machines...) and the bloat
> is going to
> >> be considerable.
> >Luigi will reply to this one.
>
> Here I am. Of course you're right about compiler support (we already had
> quite a few headaches because of this) and about the bloat. I am
> also aware
> that, as Peter Schmitteckert wrote,
>
> >for a library, I prefer virtual classes/functions to achieve conveniency
> >and flexibility. You can extend classes with virtual function, withouot
> >recompiling or changing the interface. With templates you can't
> (normally) not.
> >...and remember, virtual fuctions are just a pointer-derefencation, and
> >that's cheap on microprocessors.
>
> Well, I'll try and explain the reasoning behind the current template
> implementation, of which-I think-the finite difference package is
> the most
> blatant example. (P.S. Sorry - it turned out to be a long e-mail, but I
> wanted to put all my cards on the table so that we can continue
> discussing
> the issue with the complete picture in mind)
>
> First and foremost, the virtual classes approach is one that we
> definitely
> want to pursue - expecially as we are exporting the library
> towards Python
> (and hopefully other languages) where we cannot exploit the C++ template
> mechanism.
> In short, I surely want to be able to use dynamic polymorphism to switch
> between FiniteDifferenceModel<CrankNicolson<TridiagonalOperator> >,
> FiniteDifferenceModel<BackwardEuler<MultigridOperator> >,
> FiniteDifferenceModel<MyVerySpecificPadeApproximation>,
> FiniteDifferenceModel<WhateverYouLike>, and even
> TreeModel<WhateverItNeeds>.
> This can be (more or less) easily accomplished by inheriting
> template<class
> Evolver> FiniteDifferenceModel<Evolver> from some more generic interface
> class. However, this is a part of the global design that didn't make into
> the library yet. We just uploaded the inner part of it.
>
> At the same time, while very flexible with respect to usability by the
> final user, dynamic polymorphism is somehow less flexible with respect to
> developing new classes - because it forces such classes to belong to a
> particular hierarchy. I see two main disadvantages of this approach:
> 1) multiple inheritance rears its ugly head, which I would avoid if
> possible. This is very likely to be a only a prejudice of mine - so feel
> free to ignore it - but it is getting much too close to virtual base
> classes for my liking.
> 2) if you decide that, e.g., some sparse array class you found in another
> library would be just right for your particular problem, you'll
> have to either:
>    a) modify that class so that it inherits from the base class
> for arrays
> in our brand new polymorhpic framework, which is not always possible, or
>    b) write a wrapper class which contains the sparse array and inherit
> from the base one.
> In a template framework, you can just go ahead and write your
> differential
> operator using the sparse array class as it is, and its full
> compatibility
> with the framework is just a trait specialization away.
>
> Furthermore, if different array classes - possibly including
> sparse ones -
> or different operator classes  - tridiagonal, band diagonal, sparse, you
> name it - are to be derived from base classes, then it is very
> likely that
> we will have to declare as virtual even very basic functionalities (i.e.,
> operator[]) which tend to get in the innermost loops of the code and get
> called humongous number of times. Now, while it is true that
> virtual calls
> are cheap nowadays, it is also true that precious few optimizers inline
> them. A simple algorithm for multiplying a matrix and a vector
> would either:
> a) be written in a generic way, in which case it would jump in
> and out the
> respective operator[] for most of the time, which will add up to quite a
> loss in efficiency, or
> b) be specialized for the concrete operators and array - however, this
> would need double dispatching, which is not easier to write and maintain
> than templates.
>
> Finally, there's still one thing that static polymorphism can do and
> dynamic polymorphism can't, and that's generic algorithms on
> iterators. You
> simply can't derive all iterators from a single class as Java
> does, because
> you can't add a parent class to C pointers. But I don't think you need to
> hear this from me :)
>
> Well, I'm open to suggestions. In short, my point was: I'd like
> to be able
> to use templates in the inner parts of the models - arrays,
> operators... -
> for the kind of flexibility I explained above. I'd also like to
> use dynamic
> polymorphism in the outer parts - the complete models and their
> interface -
> for the kind of flexibility users, including myself, want.
> Once the design is refined, we could have the best of both worlds
> - and of
> course a bit of the worst of both, compiler support included :)
> But it is a
> trade-off that I would personally accept.
>
> Thanks for listening (man, was this long!)
>
>                  Luigi
>
>
> _______________________________________________
> Quantlib-users mailing list
> [hidden email]
> http://lists.sourceforge.net/mailman/listinfo/quantlib-users
>

[Bernd Johannes Wuebben-2]

Hi Luigi,
        We have all read the text books and now about the
conceptual advanges of templates. If I could I would use them
all day, but I don't and that is for issues of practical software
engineering not because I am not fully aware of the many
advantages they afford from a theoretical point of view.
What I am concerned about is

a) performance hit
b) significant bloat
c) considerable porting headaches.
d) diminished transparence as only few understand templates well
    enough.

        These issues must be weight against the beauty of
programming using templates. In the end I tend to decide to
do what is best for the user, and not what is most convenient
for the programmer. I am particularly concerned by d), as it is
hard enough to find people who know how to write pricers, it is
even harder to find those who also are well versed in C++ and
templates. The use of templates  will raise the barriers of entry
amoung students, academics and practitioners alike.

        However, this is just my opinion and nothing more,
you guys should do what you like best, this is a free software
project after all, and we want to have fun. However, I just
thought I  should make sure that we realize  what the trade
off's are.

Best,
Bernd


On Tuesday 19 December 2000 10:50, you wrote:

> At 07:18 PM 12/18/00 +0100, Ferdinando Ametrano wrote:
> >>Initial impressions:
> >>
> >>         I am unhappy about the heavy use of templates and the stl. Its
> >> going to bite you on many platforms (guys there is a whole world out
> >> there apart from Macs and Windows machines...) and the bloat is going to
> >> be considerable.
> >
> >Luigi will reply to this one.
>
> Here I am. Of course you're right about compiler support (we already had
> quite a few headaches because of this) and about the bloat. I am also aware
> that, as Peter Schmitteckert wrote,
>
> >for a library, I prefer virtual classes/functions to achieve conveniency
> >and flexibility. You can extend classes with virtual function, withouot
> >recompiling or changing the interface. With templates you can't (normally)
> > not. ...and remember, virtual fuctions are just a pointer-derefencation,
> > and that's cheap on microprocessors.
>
> Well, I'll try and explain the reasoning behind the current template
> implementation, of which-I think-the finite difference package is the most
> blatant example. (P.S. Sorry - it turned out to be a long e-mail, but I
> wanted to put all my cards on the table so that we can continue discussing
> the issue with the complete picture in mind)
>
> First and foremost, the virtual classes approach is one that we definitely
> want to pursue - expecially as we are exporting the library towards Python
> (and hopefully other languages) where we cannot exploit the C++ template
> mechanism.
> In short, I surely want to be able to use dynamic polymorphism to switch
> between FiniteDifferenceModel<CrankNicolson<TridiagonalOperator> >,
> FiniteDifferenceModel<BackwardEuler<MultigridOperator> >,
> FiniteDifferenceModel<MyVerySpecificPadeApproximation>,
> FiniteDifferenceModel<WhateverYouLike>, and even
> TreeModel<WhateverItNeeds>. This can be (more or less) easily accomplished
> by inheriting template<class Evolver> FiniteDifferenceModel<Evolver> from
> some more generic interface class. However, this is a part of the global
> design that didn't make into the library yet. We just uploaded the inner
> part of it.
>
> At the same time, while very flexible with respect to usability by the
> final user, dynamic polymorphism is somehow less flexible with respect to
> developing new classes - because it forces such classes to belong to a
> particular hierarchy. I see two main disadvantages of this approach:
> 1) multiple inheritance rears its ugly head, which I would avoid if
> possible. This is very likely to be a only a prejudice of mine - so feel
> free to ignore it - but it is getting much too close to virtual base
> classes for my liking.
> 2) if you decide that, e.g., some sparse array class you found in another
> library would be just right for your particular problem, you'll have to
> either: a) modify that class so that it inherits from the base class for
> arrays in our brand new polymorhpic framework, which is not always
> possible, or b) write a wrapper class which contains the sparse array and
> inherit from the base one.
> In a template framework, you can just go ahead and write your differential
> operator using the sparse array class as it is, and its full compatibility
> with the framework is just a trait specialization away.
>
> Furthermore, if different array classes - possibly including sparse ones -
> or different operator classes  - tridiagonal, band diagonal, sparse, you
> name it - are to be derived from base classes, then it is very likely that
> we will have to declare as virtual even very basic functionalities (i.e.,
> operator[]) which tend to get in the innermost loops of the code and get
> called humongous number of times. Now, while it is true that virtual calls
> are cheap nowadays, it is also true that precious few optimizers inline
> them. A simple algorithm for multiplying a matrix and a vector would
> either: a) be written in a generic way, in which case it would jump in and
> out the respective operator[] for most of the time, which will add up to
> quite a loss in efficiency, or
> b) be specialized for the concrete operators and array - however, this
> would need double dispatching, which is not easier to write and maintain
> than templates.
>
> Finally, there's still one thing that static polymorphism can do and
> dynamic polymorphism can't, and that's generic algorithms on iterators. You
> simply can't derive all iterators from a single class as Java does, because
> you can't add a parent class to C pointers. But I don't think you need to
> hear this from me :)
>
> Well, I'm open to suggestions. In short, my point was: I'd like to be able
> to use templates in the inner parts of the models - arrays, operators... -
> for the kind of flexibility I explained above. I'd also like to use dynamic
> polymorphism in the outer parts - the complete models and their interface -
> for the kind of flexibility users, including myself, want.
> Once the design is refined, we could have the best of both worlds - and of
> course a bit of the worst of both, compiler support included :) But it is a
> trade-off that I would personally accept.
>
> Thanks for listening (man, was this long!)
>
>                  Luigi
>
>
> _______________________________________________
> Quantlib-users mailing list
> [hidden email]
> http://lists.sourceforge.net/mailman/listinfo/quantlib-users

[Luigi Ballabio-3]

Hi Bernd (and all the others),

At 03:26 PM 12/19/00 -0500, you wrote:

>Hi Luigi,
>         We have all read the text books and now about the
>conceptual advanges of templates.

Ok, I'll keep it shorter next time :)

>What I am concerned about is
>
>a) performance hit

This is a pro for templates.

>b) significant bloat

And this is a con. They cancel out in my opinion.

>c) considerable porting headaches.

This should be less of a problem as time goes by and compilers improve.
We're talking standard C++ after all... and anyway, I'm afraid the
headaches will be more on the Win side - I'm told g++ is well ahead of
other competitors as far as compliance goes. However: I'm aware of the
problem, and of the fact the compilers won't uniform for a while. But I
don't think there is an infinite number of possibilities to mess things up
- I mean, I think we'll have a relatively small number of generic
techniques to fix porting problems, so that it shouldn't be hard to master
them.

>d) diminished transparence as only few understand templates well
>     enough.
>
>         These issues must be weight against the beauty of
>programming using templates. In the end I tend to decide to
>do what is best for the user, and not what is most convenient
>for the programmer. I am particularly concerned by d), as it is
>hard enough to find people who know how to write pricers, it is
>even harder to find those who also are well versed in C++ and
>templates. The use of templates  will raise the barriers of entry
>amoung students, academics and practitioners alike.

I agree. But as I said: there's only half of the framework in place now.
The outer layer will be a base class (let us say Model) from which
FiniteDifferenceModel inherits. Pricers will be written in terms of the
Model interface, which means that one can use either our finite difference
framework and benefit from it if he can, or write his own model class
inherited from Model without using templates. You're free to choose what
techniques to use: the power and complexity of templates, or the ease of
dynamic polymorphism. The _complete_ framework will accept either one.

I am aware that this is the second mail in which I call to my rescue this
supposed complete framework which nobody ever saw. Well, I must admit I
didn't see it either :) Unfortunately we have a few deadlines at the
moment, so that I don't have much time to set it up. However, I hope I did
make the basic idea clear, so that it can be taken into account.

Thanks again for your patience,

                         Luigi

[Ferdinando M. Ametrano-2]

Hi Bernd and all

>d) diminished transparence as only few understand templates well
>     enough.
>I am particularly concerned by d), as it is
>hard enough to find people who know how to write pricers, it is
>even harder to find those who also are well versed in C++ and
>templates. The use of templates  will raise the barriers of entry
>amoung students, academics and practitioners alike.
We've been through this discussion a couple of time in RiskMap.
We (finally?) agreed that QuantLib should be a 3 level project:
1) core base
2) pricing code
3) end user interface

I will give an (incomplete) example. Let's consider this first level statement:
FiniteDifferenceModel<CrankNicolson<TridiagonalOperator> >
myModel(theOperator);

[where theOperator is the tridiagonal discretization of BS equation]

It's scaring, I agree. The presence of that line of code in
bsmamericanoption.cpp is highly unsatisfactory. This is because the second
level is not ready yet.
The second level design is sketched in the same bsmamericanoption.cpp.
We will have a Model class implementing a common interface (rollBack(),
etc.) and somewhere in the first level there will be:
template <class Evolver>
class FiniteDifferenceModel : public Model {
}

typedef FiniteDifferenceModel<CrankNicolson<TridiagonalOperator> > BsmModel;

At this point, you can declare
Handle<Model> myModel(new BsmModel(gridSize));
[the safe equivalent of:
  Model* myModel = new BsmModel(gridSize);
]

Now - in the second level - the Model interface will remove all
implementation details (binomial, trinomial, finite difference) away and
will provide a common interface for any model. This will work for BDT, BK,
HW, etc. too, giving you the ability to write pricing code with pluggable
Model. Switching from one model to another will be very easy.
No template programming will be needed either.

The third level is the end user interface. Here I'm thinking about
scripting languages, MATLAB, Octave, Excel, applications that can be
written on QuantLib.
We will have to design an object interface for the environments supporting
objects and function interface for the others.
In the third level should be still possible to write pricing code (I'm
especially thinking of Python): of course this code will be inefficient,
but good enough for prototyping or not-demanding application.

ciao -- Nando

[Peter Schmitteckert-2]

Salut Gilbert,

On Die, 19 Dez 2000, you wrote:

>Some headaches I had using templates:
>A few months ago, I wrote some C++ template code using STL templates (which
>in turn contained nested templates). I had problems looking at the value of
>the variables in my Visual C++ debugger because some of the templates were
>resolved into over 20 lines or so. Perhaps I didn't find the option that
>would allow me to collapse the expanded template name into something more
>managable. If there isn't, then templates can make your debugging life much
>more difficult.
>Regards
>Gilbert

I normally create pointer - debug-variables poiting to &(*iterator).
Then you can access the members. Annoying, but it works.

Or, although I'm developping a lot in MS VS 6.0, I often port
it my sources to GCC/Linux fjust or Debugging templated Code with DDD.

Best wishes
Peter

[Peter Schmitteckert-2]

Hello,

>At 07:18 PM 12/18/00 +0100, Ferdinando Ametrano wrote:
[...]

>Furthermore, if different array classes - possibly including sparse ones -
>or different operator classes  - tridiagonal, band diagonal, sparse, you
>name it - are to be derived from base classes, then it is very likely that
>we will have to declare as virtual even very basic functionalities (i.e.,
>operator[]) which tend to get in the innermost loops of the code and get
>called humongous number of times. Now, while it is true that virtual calls
>are cheap nowadays, it is also true that precious few optimizers inline
>them. A simple algorithm for multiplying a matrix and a vector would either:
>a) be written in a generic way, in which case it would jump in and out the
>respective operator[] for most of the time, which will add up to quite a
>loss in efficiency, or
>b) be specialized for the concrete operators and array - however, this
>would need double dispatching, which is not easier to write and maintain
>than templates.

Yes, I agree, that expression templates are a nice and efficient way for
Linear algebra. But, there the rules are simple: You need an () or [] operator.
And I do like the STL library.

But I have to agree to Bernd Wuebbens point c):
> c) considerable porting headaches.

In 1997 had heavily templated sources which went to SGI, NEC, Cray just because
their compiler couldn't compile my sources. SGI (and now Cray) works now.
MS C++ 6.0 and gcc 2.95.2 still can't.
If you ever want to run your library on a dinosauror (large vector machines),
heavily templated code will kill you, or wait 10 years for their compilers to
catch up. That's  just a hint.
The main problem is partial template specialization and the non-generic
template instantiation mechanism.

Best wishes
Peter

 ________________________________________
Dr. Peter Schmitteckert / IT-Consulting
s-mail: Fridolinstr. 19, 68753 Waghäusel
Web: http://www.schmitteckert.com
e-mail: [hidden email]
Tel.: [phone]

[Ferdinando Ametrano-6]

Peter

I'm unable to reach you at [hidden email]

please get in touch

ciao -- Nando