[GNC] OFX import sometimes loses transaction memos

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[GNC] OFX import sometimes loses transaction memos

David Carlson-4
Using GnuCash 2.6.17? in Ubunutu 16.04 I noticed that the OFX importer from
time to time will fail to import some, but not all transaction memos for
only one of the banks that that are currently active in my data.

This bank tends to send lenghthy memos.  One of the failed memos had 98
ASCII characters, another accepted memo had 42 ASCII characters.  The OFX
files from this bank do not contain cr or lf characters before the <
characters, instead have 15 spaces.

Does anyone else have. A similar problem or a suggestion about might cause
this?


David C
_______________________________________________
gnucash-user mailing list
[hidden email]
To update your subscription preferences or to unsubscribe:
https://lists.gnucash.org/mailman/listinfo/gnucash-user
If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
-----
Please remember to CC this list on all your replies.
You can do this by using Reply-To-List or Reply-All.
Reply | Threaded
Open this post in threaded view
|

Re: [GNC] OFX import sometimes loses transaction memos

David Cousens
David

Have you tried examining the file with ofxdump? If you are on Linux, it can
be installed with "sudo apt install ofx".  Not sure about how to get it for
WIndows or MacOS. It will dump to stdout whatever ofxdump can determine
about an ofx response file including error messages. It tests conformance
with the libofx librarryofxdump. You may need to use it with the debug
switch and possibly the parsing messages if using it without the optional
switches doesn't give enough info. I.e.

$ofxdump --msg_debug --msg_parser<filename>

David Cousens



-----
David Cousens
--
Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html
_______________________________________________
gnucash-user mailing list
[hidden email]
To update your subscription preferences or to unsubscribe:
https://lists.gnucash.org/mailman/listinfo/gnucash-user
If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
-----
Please remember to CC this list on all your replies.
You can do this by using Reply-To-List or Reply-All.
David Cousens
Reply | Threaded
Open this post in threaded view
|

Re: [GNC] OFX import sometimes loses transaction memos

David Carlson-4
On Wed, Jan 16, 2019, 4:36 PM David Cousens <[hidden email] wrote:

> David
>
> Have you tried examining the file with ofxdump? If you are on Linux, it can
> be installed with "sudo apt install ofx".  Not sure about how to get it for
> WIndows or MacOS. It will dump to stdout whatever ofxdump can determine
> about an ofx response file including error messages. It tests conformance
> with the libofx librarryofxdump. You may need to use it with the debug
> switch and possibly the parsing messages if using it without the optional
> switches doesn't give enough info. I.e.
>
> $ofxdump --msg_debug --msg_parser<filename>
>
> David Cousens
>
>
>
> -----
> David Cousens
> --
> Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html
> _______________________________________________
> gnucash-user mailing list
> [hidden email]
> To update your subscription preferences or to unsubscribe:
> https://lists.gnucash.org/mailman/listinfo/gnucash-user
> If you are using Nabble or Gmane, please see
> https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
> -----
> Please remember to CC this list on all your replies.
> You can do this by using Reply-To-List or Reply-All.
>

David,

I have already examined one of the OFX files that has a mix of some trans
actions with memos that were and we're not imported with a binary editor
and found no difference except possibly number of characters in the string.

Right now I am away from home and cannot try OFX DUMP to see if it knows
whether there is a maximum string length, but if there is and it is less
than 200 characters I would be surprised.

If that is true, I would hope that it would truncate strings rather than
drop them.

David C arlson

>
_______________________________________________
gnucash-user mailing list
[hidden email]
To update your subscription preferences or to unsubscribe:
https://lists.gnucash.org/mailman/listinfo/gnucash-user
If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
-----
Please remember to CC this list on all your replies.
You can do this by using Reply-To-List or Reply-All.
Reply | Threaded
Open this post in threaded view
|

Re: [GNC] OFX import sometimes loses transaction memos

David Cousens
David

Agreed. The OFX format should in principle have no limitation on a string length as it should be anything between
opening and closing tags, but there may be on importation in GnuCash.  Not familiar enough with the OFX importer to be
sure whether it imposes any restrictions or not at this stage.

David Cousens
On Wed, 2019-01-16 at 18:19 -0600, David Carlson wrote:

>
>
> On Wed, Jan 16, 2019, 4:36 PM David Cousens <[hidden email] wrote:
> > David
> >
> >
> >
> > Have you tried examining the file with ofxdump? If you are on Linux, it can
> >
> > be installed with "sudo apt install ofx".  Not sure about how to get it for
> >
> > WIndows or MacOS. It will dump to stdout whatever ofxdump can determine
> >
> > about an ofx response file including error messages. It tests conformance
> >
> > with the libofx librarryofxdump. You may need to use it with the debug
> >
> > switch and possibly the parsing messages if using it without the optional
> >
> > switches doesn't give enough info. I.e.
> >
> >
> >
> > $ofxdump --msg_debug --msg_parser<filename>
> >
> >
> >
> > David Cousens
> >
> >
> >
> >
> >
> >
> >
> > -----
> >
> > David Cousens
> >
> > --
> >
> > Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html
> >
> > _______________________________________________
> >
> > gnucash-user mailing list
> >
> > [hidden email]
> >
> > To update your subscription preferences or to unsubscribe:
> >
> > https://lists.gnucash.org/mailman/listinfo/gnucash-user
> >
> > If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
> >
> > -----
> >
> > Please remember to CC this list on all your replies.
> >
> > You can do this by using Reply-To-List or Reply-All.
>
> David,
>
> I have already examined one of the OFX files that has a mix of some trans actions with memos that were and we're not
> imported with a binary editor and found no difference except possibly number of characters in the string.
>
> Right now I am away from home and cannot try OFX DUMP to see if it knows whether there is a maximum string length, but
> if there is and it is less than 200 characters I would be surprised.
>
> If that is true, I would hope that it would truncate strings rather than drop them.
>
> David C arlson
>
>
_______________________________________________
gnucash-user mailing list
[hidden email]
To update your subscription preferences or to unsubscribe:
https://lists.gnucash.org/mailman/listinfo/gnucash-user
If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
-----
Please remember to CC this list on all your replies.
You can do this by using Reply-To-List or Reply-All.
David Cousens
Reply | Threaded
Open this post in threaded view
|

Re: [GNC] OFX import sometimes loses transaction memos

GnuCash - User mailing list
The OFX 2.1.1 standard limits <MEMO> elements to a maximum of 255 alphanumeric characters. Today’s principles are different from those of a couple (or more) decades ago when these standards were started and both memory and storage were orders of magnitude more expensive than they are now.


--
Dave Reiser
[hidden email]





> On Jan 16, 2019, at 11:09 PM, David Cousens <[hidden email]> wrote:
>
> David
>
> Agreed. The OFX format should in principle have no limitation on a string length as it should be anything between
> opening and closing tags, but there may be on importation in GnuCash.  Not familiar enough with the OFX importer to be
> sure whether it imposes any restrictions or not at this stage.
>
> David Cousens
> On Wed, 2019-01-16 at 18:19 -0600, David Carlson wrote:
>>
>>
>> On Wed, Jan 16, 2019, 4:36 PM David Cousens <[hidden email] wrote:
>>> David
>>>
>>>
>>>
>>> Have you tried examining the file with ofxdump? If you are on Linux, it can
>>>
>>> be installed with "sudo apt install ofx".  Not sure about how to get it for
>>>
>>> WIndows or MacOS. It will dump to stdout whatever ofxdump can determine
>>>
>>> about an ofx response file including error messages. It tests conformance
>>>
>>> with the libofx librarryofxdump. You may need to use it with the debug
>>>
>>> switch and possibly the parsing messages if using it without the optional
>>>
>>> switches doesn't give enough info. I.e.
>>>
>>>
>>>
>>> $ofxdump --msg_debug --msg_parser<filename>
>>>
>>>
>>>
>>> David Cousens
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -----
>>>
>>> David Cousens
>>>
>>> --
>>>
>>> Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html
>>>
>>> _______________________________________________
>>>
>>> gnucash-user mailing list
>>>
>>> [hidden email]
>>>
>>> To update your subscription preferences or to unsubscribe:
>>>
>>> https://lists.gnucash.org/mailman/listinfo/gnucash-user
>>>
>>> If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
>>>
>>> -----
>>>
>>> Please remember to CC this list on all your replies.
>>>
>>> You can do this by using Reply-To-List or Reply-All.
>>
>> David,
>>
>> I have already examined one of the OFX files that has a mix of some trans actions with memos that were and we're not
>> imported with a binary editor and found no difference except possibly number of characters in the string.
>>
>> Right now I am away from home and cannot try OFX DUMP to see if it knows whether there is a maximum string length, but
>> if there is and it is less than 200 characters I would be surprised.
>>
>> If that is true, I would hope that it would truncate strings rather than drop them.
>>
>> David C arlson
>>
>>
> _______________________________________________
> gnucash-user mailing list
> [hidden email]
> To update your subscription preferences or to unsubscribe:
> https://lists.gnucash.org/mailman/listinfo/gnucash-user
> If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
> -----
> Please remember to CC this list on all your replies.
> You can do this by using Reply-To-List or Reply-All.

_______________________________________________
gnucash-user mailing list
[hidden email]
To update your subscription preferences or to unsubscribe:
https://lists.gnucash.org/mailman/listinfo/gnucash-user
If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
-----
Please remember to CC this list on all your replies.
You can do this by using Reply-To-List or Reply-All.
Reply | Threaded
Open this post in threaded view
|

Re: [GNC] OFX import sometimes loses transaction memos

David Carlson-4
I think that I need to spend more time examining the transactions that had
their memos dropped during import to see if I can find a threshold string
length, then file a bug report asking for a short term fix truncating
strings rather than dropping them, and a long term fix accepting up to the
256 character max and truncating after that.

David C

On Wed, Jan 16, 2019 at 10:44 PM David Reiser <[hidden email]> wrote:

> The OFX 2.1.1 standard limits <MEMO> elements to a maximum of 255
> alphanumeric characters. Today’s principles are different from those of a
> couple (or more) decades ago when these standards were started and both
> memory and storage were orders of magnitude more expensive than they are
> now.
>
>
> --
> Dave Reiser
> [hidden email]
>
>
>
>
>
> > On Jan 16, 2019, at 11:09 PM, David Cousens <[hidden email]>
> wrote:
> >
> > David
> >
> > Agreed. The OFX format should in principle have no limitation on a
> string length as it should be anything between
> > opening and closing tags, but there may be on importation in GnuCash.
> Not familiar enough with the OFX importer to be
> > sure whether it imposes any restrictions or not at this stage.
> >
> > David Cousens
> > On Wed, 2019-01-16 at 18:19 -0600, David Carlson wrote:
> >>
> >>
> >> On Wed, Jan 16, 2019, 4:36 PM David Cousens <[hidden email]
> wrote:
> >>> David
> >>>
> >>>
> >>>
> >>> Have you tried examining the file with ofxdump? If you are on Linux,
> it can
> >>>
> >>> be installed with "sudo apt install ofx".  Not sure about how to get
> it for
> >>>
> >>> WIndows or MacOS. It will dump to stdout whatever ofxdump can determine
> >>>
> >>> about an ofx response file including error messages. It tests
> conformance
> >>>
> >>> with the libofx librarryofxdump. You may need to use it with the debug
> >>>
> >>> switch and possibly the parsing messages if using it without the
> optional
> >>>
> >>> switches doesn't give enough info. I.e.
> >>>
> >>>
> >>>
> >>> $ofxdump --msg_debug --msg_parser<filename>
> >>>
> >>>
> >>>
> >>> David Cousens
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> -----
> >>>
> >>> David Cousens
> >>>
> >>> --
> >>>
> >>> Sent from:
> http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html
> >>>
> >>> _______________________________________________
> >>>
> >>> gnucash-user mailing list
> >>>
> >>> [hidden email]
> >>>
> >>> To update your subscription preferences or to unsubscribe:
> >>>
> >>> https://lists.gnucash.org/mailman/listinfo/gnucash-user
> >>>
> >>> If you are using Nabble or Gmane, please see
> https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
> >>>
> >>> -----
> >>>
> >>> Please remember to CC this list on all your replies.
> >>>
> >>> You can do this by using Reply-To-List or Reply-All.
> >>
> >> David,
> >>
> >> I have already examined one of the OFX files that has a mix of some
> trans actions with memos that were and we're not
> >> imported with a binary editor and found no difference except possibly
> number of characters in the string.
> >>
> >> Right now I am away from home and cannot try OFX DUMP to see if it
> knows whether there is a maximum string length, but
> >> if there is and it is less than 200 characters I would be surprised.
> >>
> >> If that is true, I would hope that it would truncate strings rather
> than drop them.
> >>
> >> David C arlson
> >>
> >>
> > _______________________________________________
> > gnucash-user mailing list
> > [hidden email]
> > To update your subscription preferences or to unsubscribe:
> > https://lists.gnucash.org/mailman/listinfo/gnucash-user
> > If you are using Nabble or Gmane, please see
> https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
> > -----
> > Please remember to CC this list on all your replies.
> > You can do this by using Reply-To-List or Reply-All.
>
>
_______________________________________________
gnucash-user mailing list
[hidden email]
To update your subscription preferences or to unsubscribe:
https://lists.gnucash.org/mailman/listinfo/gnucash-user
If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
-----
Please remember to CC this list on all your replies.
You can do this by using Reply-To-List or Reply-All.
Reply | Threaded
Open this post in threaded view
|

Re: [GNC] OFX import sometimes loses transaction memos

Geert Janssens-4
Op donderdag 17 januari 2019 06:33:25 CET schreef David Carlson:
> I think that I need to spend more time examining the transactions that had
> their memos dropped during import to see if I can find a threshold string
> length, then file a bug report asking for a short term fix truncating
> strings rather than dropping them, and a long term fix accepting up to the
> 256 character max and truncating after that.
>
> David C

I don't think there should be one. Some time back I submitted a number of
patches to handle long strings in the ofx library. IIRC it internally uses a
fixed-length buffer to read parts of the ofx file. And I believe that buffer
itself was already 1024 characters long. So at least for lines (including
tags) of that size there should not be a problem. Next if the buffer gets full
before a full tag contents could be read, it simply fills that buffer a second
time and appends that contents to the same first string. So in principle you
can have text messages of unlimited size.

Now there were bugs in that area, and my patches were partly to resolve those.
IIRC one issue was if the ofx file didn't have newlines between tags that
buffer handling would not always work as it should.

Note those changes have gone into a recent libofx release. As it's been a
while I don't know whether gnucash 2.6.x was ever packaged for Windows with
this more recent release though I'm sure gnucash 3.x is.

And of course there may be other bugs that weren't reported yet. I'm not using
ofx myself.

Lastly it's probably still interesting to run ofxdump as it can help determine
whether the issue already starts in the libofx library or rather that gnucash
messes up the data it received from libofx.

Regards,

Geert


_______________________________________________
gnucash-user mailing list
[hidden email]
To update your subscription preferences or to unsubscribe:
https://lists.gnucash.org/mailman/listinfo/gnucash-user
If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
-----
Please remember to CC this list on all your replies.
You can do this by using Reply-To-List or Reply-All.
Reply | Threaded
Open this post in threaded view
|

Re: [GNC] OFX import sometimes loses transaction memos

David Carlson-4
On Thu, Jan 17, 2019, 1:15 AM Geert Janssens <[hidden email]
wrote:

> Op donderdag 17 januari 2019 06:33:25 CET schreef David Carlson:
> > I think that I need to spend more time examining the transactions that
> had
> > their memos dropped during import to see if I can find a threshold string
> > length, then file a bug report asking for a short term fix truncating
> > strings rather than dropping them, and a long term fix accepting up to
> the
> > 256 character max and truncating after that.
> >
> > David C
>
> I don't think there should be one. Some time back I submitted a number of
> patches to handle long strings in the ofx library. IIRC it internally uses
> a
> fixed-length buffer to read parts of the ofx file. And I believe that
> buffer
> itself was already 1024 characters long. So at least for lines (including
> tags) of that size there should not be a problem. Next if the buffer gets
> full
> before a full tag contents could be read, it simply fills that buffer a
> second
> time and appends that contents to the same first string. So in principle
> you
> can have text messages of unlimited size.
>
> Now there were bugs in that area, and my patches were partly to resolve
> those.
> IIRC one issue was if the ofx file didn't have newlines between tags that
> buffer handling would not always work as it should.
>
> Note those changes have gone into a recent libofx release. As it's been a
> while I don't know whether gnucash 2.6.x was ever packaged for Windows
> with
> this more recent release though I'm sure gnucash 3.x is.
>
> And of course there may be other bugs that weren't reported yet. I'm not
> using
> ofx myself.
>
> Lastly it's probably still interesting to run ofxdump as it can help
> determine
> whether the issue already starts in the libofx library or rather that
> gnucash
> messes up the data it received from libofx.
>
> Regards,
>
> Geert
>
>
>
I was not aware that release 3.4 might behave better.  I have not tried it
yet for other reasons.  This is good news.

I think that  i mentioned that my test files do not have carriage returns,
line feeds or newlines, instead preceeding the less than symbols with 15
space characters.
I did not mention that they also contain transactions belonging to several
different bank accounts within the same OFX file.

I shall be sure to include a sanitized test file when I submit my bug
report.

Aside:  I have discovered quite by accident that the keyboard on this
tablet has a Ctrl key that actually works to help manipulate a buffer.  It
is still tricky to insert text after or in the middle of the previous
reply.  I think it worked! 😊

Now if I can learn how to override the spell checker when it is wrong....

David C



>
_______________________________________________
gnucash-user mailing list
[hidden email]
To update your subscription preferences or to unsubscribe:
https://lists.gnucash.org/mailman/listinfo/gnucash-user
If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
-----
Please remember to CC this list on all your replies.
You can do this by using Reply-To-List or Reply-All.