Extending QLXL and serialization issue

[Lapin]

Hi,

I have extended QLXL but I have an issue with the serialization.
When I try to deserialize the object that were created befate the update I get a stream error (more or less the old object is not recognized).

I have looked into it, and I have found the reason why. When I have added the new classes, the class_id="251" in the xml object is not the same as in the previous object. For example, I get the following:

Old object:
<px class_id="149" tracking_level="1" version="0" object_id="_90">
<ObjectId>GBPYC6M_MID</ObjectId>
<ClassName>qlPiecewiseYieldCurve</ClassName>

New object:
<px class_id="153" tracking_level="1" version="0" object_id="_90">
<ObjectId>GBPYC6M_MID</ObjectId>
<ClassName>qlPiecewiseYieldCurve</ClassName>

I have not touched to this object nor I have touched any related part to the yield curve term structure object.

If in the xml I put back the class_id as it was before the change (addind 4 to each of them in the xml file), the deserialization works just fine.

I was unable to locate where we can control this behavior or just why it is doing so.

Thanks

[Ballabio Gerardo-4]

I have run into the same issue.

In the generated xml files, every QLXL object is identified by a reference number. If I understand correctly, the numbers are defined by the order in which object are registered in the functions called by registerQlFunctions (see QuantLibXL/qlxl/register/register_all.cpp and other files in that directory). Whenever you insert a new object in the middle of the sequence, all subsequent objects get their numbers incremented by one.

In order to avoid that problem, new objects should only be added at the end of the sequence. But because the functions which do the registering are autogenerated by gensrc which, if I am not mistaken, sorts objects first by category and then alphabetically, this is not doable. You'll have to live with that.

I'd suggest that you write a utility program to "convert" your files to the new numbering sequence. That's the only way to port them between different QLXL releases or development versions.
 
Gerardo


-----Messaggio originale-----
Da: Lapin [mailto:[hidden email]]
Inviato: venerdì 11 gennaio 2013 16.54
A: [hidden email]
Oggetto: [Quantlib-users] Extending QLXL and serialization issue

Hi,

I have extended QLXL but I have an issue with the serialization.
When I try to deserialize the object that were created befate the update I get a stream error (more or less the old object is not recognized).

I have looked into it, and I have found the reason why. When I have added the new classes, the class_id="251" in the xml object is not the same as in the previous object. For example, I get the following:

*Old object*:
<px class_id="149" tracking_level="1" version="0" object_id="_90"> <ObjectId>GBPYC6M_MID</ObjectId> <ClassName>qlPiecewiseYieldCurve</ClassName>

*New object*:
<px class_id="153" tracking_level="1" version="0" object_id="_90"> <ObjectId>GBPYC6M_MID</ObjectId> <ClassName>qlPiecewiseYieldCurve</ClassName>

I have not touched to this object nor I have touched any related part to the yield curve term structure object.

If in the xml I put back the class_id as it was before the change (addind 4 to each of them in the xml file), the deserialization works just fine.

I was unable to locate where we can control this behavior or just why it is doing so.

Thanks




--
View this message in context: http://quantlib.10058.n7.nabble.com/Extending-QLXL-and-serialization-issue-tp13901.html
Sent from the quantlib-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Master HTML5, CSS3, ASP.NET, MVC, AJAX, Knockout.js, Web API and much more. Get web development skills now with LearnDevNow -
350+ hours of step-by-step video tutorials by Microsoft MVPs and experts.
SALE $99.99 this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122812
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head></head>
<body>
<div style="font-family:Calibri;font-size:10px">
Banca Profilo S.p.A.
Corso Italia, 49 - 20122 Milano - Tel. [phone], Fax [phone]
Capitale Sociale Euro 136.794.106,00 i.v.
Iscrizione al Registro Imprese di Milano, C.F. e P.IVA 09108700155 - [hidden email]
Iscritta all’Albo delle Banche e dei Gruppi bancari
Aderente al Fondo Interbancario di Tutela dei depositi
Aderente al Conciliatore Bancario Finanziario e all’Arbitro Bancario Finanziario
Appartenente al Gruppo bancario Banca Profilo e soggetta all’attività di direzione e coordinamento di Arepo BP S.p.A.


DISCLAIMER:
The information transmitted may contain confidential and/or privileged material.
Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon,
this information by persons or entities other than the intended recipient is prohibited.
If you received this in error, please contact the sender and delete the material from any computer.
</div>
</body>
</html>


------------------------------------------------------------------------------
Master HTML5, CSS3, ASP.NET, MVC, AJAX, Knockout.js, Web API and
much more. Get web development skills now with LearnDevNow -
350+ hours of step-by-step video tutorials by Microsoft MVPs and experts.
SALE $99.99 this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122812
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

[Lapin]

Hi Gerardo,

Thanks.
It looks like writing a conversion program could be quite heavy since you will have to run it on each historical file you have every time you do a change in QLXL (new model or so...).

I was wondering what could be the implcations of writing a name-value-pair kind of serialization as in the following article:

http://www.boost.org/doc/libs/1_39_0/libs/serialization/doc/wrappers.html#nvp

We would like to rewrite the generation script but do you foresee any other side effects?

Thanks

[Ballabio Gerardo-4]

Sorry, that's a question for QLXL developers to answer. I don't know enough.

Gerardo


-----Messaggio originale-----
Da: Lapin [mailto:[hidden email]]
Inviato: venerdì 11 gennaio 2013 17.41
A: [hidden email]
Oggetto: Re: [Quantlib-users] R: Extending QLXL and serialization issue

Hi Gerardo,

Thanks.
It looks like writing a conversion program could be quite heavy since you will have to run it on each historical file you have every time you do a change in QLXL (new model or so...).

I was wondering what could be the implcations of writing a name-value-pair kind of serialization as in the following article:

http://www.boost.org/doc/libs/1_39_0/libs/serialization/doc/wrappers.html#nvp

We would like to rewrite the generation script but do you foresee any other side effects?

Thanks




--
View this message in context: http://quantlib.10058.n7.nabble.com/Extending-QLXL-and-serialization-issue-tp13901p13904.html
Sent from the quantlib-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Master HTML5, CSS3, ASP.NET, MVC, AJAX, Knockout.js, Web API and much more. Get web development skills now with LearnDevNow -
350+ hours of step-by-step video tutorials by Microsoft MVPs and experts.
SALE $99.99 this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122812
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head></head>
<body>
<div style="font-family:Calibri;font-size:10px">
Banca Profilo S.p.A.
Corso Italia, 49 - 20122 Milano - Tel. [phone], Fax [phone]
Capitale Sociale Euro 136.794.106,00 i.v.
Iscrizione al Registro Imprese di Milano, C.F. e P.IVA 09108700155 - [hidden email]
Iscritta all’Albo delle Banche e dei Gruppi bancari
Aderente al Fondo Interbancario di Tutela dei depositi
Aderente al Conciliatore Bancario Finanziario e all’Arbitro Bancario Finanziario
Appartenente al Gruppo bancario Banca Profilo e soggetta all’attività di direzione e coordinamento di Arepo BP S.p.A.


DISCLAIMER:
The information transmitted may contain confidential and/or privileged material.
Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon,
this information by persons or entities other than the intended recipient is prohibited.
If you received this in error, please contact the sender and delete the material from any computer.
</div>
</body>
</html>


------------------------------------------------------------------------------
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. SALE $99.99 this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122412
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

[Eric Ehlers-2]

Hello,

On 2013-01-11 16:53, Lapin wrote:

> I have extended QLXL but I have an issue with the serialization.
> When I try to deserialize the object that were created befate the
> update I
> get a stream error (more or less the old object is not recognized).
>
> I have looked into it, and I have found the reason why. When I have
> added
> the new classes, the class_id="251" in the xml object is not the same
> as in
> the previous object.

On 2013-01-11 17:23, Ballabio Gerardo wrote:

> I have run into the same issue.
>
> In the generated xml files, every QLXL object is identified by a
> reference number. If I understand correctly, the numbers are defined
> by the order in which object are registered in the functions called
> by
> registerQlFunctions (see QuantLibXL/qlxl/register/register_all.cpp
> and
> other files in that directory). Whenever you insert a new object in
> the middle of the sequence, all subsequent objects get their numbers
> incremented by one.

That's right.  boost::serialization numbers types in the sequence in
which it encounters them.  Registering the types in alphabetical order
of group/class ensures that the assignment of IDs is deterministic at
least within any build of a given version of QuantLibAddin.

We had long debates about how to guarantee compatibility across
releases.  We considered solutions such as never deleting any types from
the serialization framework, even if the type is deprecated.  But even
then you have the problem of existing types changing and so you have to
version each type.  It's just a burden of using boost::serialization.  
In the end we did not have the resources to tackle the problem, so data
serialized from one version of the library cannot be deserialized by
another.

> I was wondering what could be the implcations of writing a
> name-value-pair
> kind of serialization as in the following article:

Well, you are talking about a nontrivial redesign, hard to give a quick
answer on that one.  Please keep us posted if you take it on.

Kind Regards,
Eric
--
===================================================
Eric Ehlers
nazcatech sprl | Brussels | http://www.nazcatech.be
* Distributed computing for pricing analytics
* Use Microsoft Excel as a client to the Grid

------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

[Ballabio Gerardo-4]

-----Messaggio originale-----
Da: Eric Ehlers [mailto:[hidden email]]
> We had long debates about how to guarantee compatibility across releases.  We considered solutions such as never deleting any types from the serialization framework, even if the type is deprecated.  But even then you have the problem of existing types changing and so you have to version each type.  It's just a burden of using boost::serialization.

Changing or deleting existing types is an API break. When you're doing that, it's reasonable to expect that you may break compatibility of serialized files too. But compatibility could at least be preserved between non-API-breaking releases.

How difficult would it be to modify gensrc so that it reads the list of types from a file and registers them in the order as they are written, instead of alphabetically? Then new types could always be added at the end of the list without changing the numbers of existing types, while deletions and reorderings would be reserved to API-breaking releases.

Gerardo


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head></head>
<body>
<div style="font-family:Calibri;font-size:10px">
Banca Profilo S.p.A.
Corso Italia, 49 - 20122 Milano - Tel. [phone], Fax [phone]
Capitale Sociale Euro 136.794.106,00 i.v.
Iscrizione al Registro Imprese di Milano, C.F. e P.IVA 09108700155 - [hidden email]
Iscritta all’Albo delle Banche e dei Gruppi bancari
Aderente al Fondo Interbancario di Tutela dei depositi
Aderente al Conciliatore Bancario Finanziario e all’Arbitro Bancario Finanziario
Appartenente al Gruppo bancario Banca Profilo e soggetta all’attività di direzione e coordinamento di Arepo BP S.p.A.


DISCLAIMER:
The information transmitted may contain confidential and/or privileged material.
Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon,
this information by persons or entities other than the intended recipient is prohibited.
If you received this in error, please contact the sender and delete the material from any computer.
</div>
</body>
</html>


------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users

[Eric Ehlers-2]

Hi Gerardo,

Apologies, I only just noticed the message that you sent in February...

On 2013-02-11 11:52, Ballabio Gerardo wrote:

> -----Messaggio originale-----
> Da: Eric Ehlers [mailto:[hidden email]]
>> We had long debates about how to guarantee compatibility across
>> releases.  We considered solutions such as never deleting any types
>> from the serialization framework, even if the type is deprecated.  But
>> even then you have the problem of existing types changing and so you
>> have to version each type.  It's just a burden of using
>> boost::serialization.
>
> Changing or deleting existing types is an API break. When you're
> doing that, it's reasonable to expect that you may break
> compatibility
> of serialized files too. But compatibility could at least be
> preserved
> between non-API-breaking releases.
>
> How difficult would it be to modify gensrc so that it reads the list
> of types from a file and registers them in the order as they are
> written, instead of alphabetically? Then new types could always be
> added at the end of the list without changing the numbers of existing
> types, while deletions and reorderings would be reserved to
> API-breaking releases.

You and I are talking about the same thing.  When I said "never
deleting any types from the serialization framework", I meant what you
said... instead of numbering the types alphabetically, we maintain a
master list of all types and we number the types in the sequence in
which they appear in that list.  The id number of each type never
changes and new types are added to the end of the list.  This would not
be hard to implement.

Kind Regards,
Eric
--
===================================================
Eric Ehlers
nazcatech sprl | Brussels | http://www.nazcatech.be
* Distributed computing for pricing analytics
* Use Microsoft Excel as a client to the Grid

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
_______________________________________________
QuantLib-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quantlib-users