Generic Import Transaction Matcher empty - revisited

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Generic Import Transaction Matcher empty - revisited

James Roman
I've read the previous thread. Unfortunately after finally getting my
system updated enough to install the gnucash-1.8.12-3.i586.rpm from
ftp.suse.com/pub/suse/i386/supplementary/GNOME/update_for_10.0/applications I still can not import OFX files from my bank.

So here are the details:
I upgraded to SuSE 10 after having the same issue with SuSe 9.3.
Intel Laptop
Gnucash version: 1.8.12
libofx version: 0.8.0-3
Numerous new applications/libraries needed to meet install requirements
for gnucahs, including aqbanking (built from source rpm), gwenhywfar
(built from source rpm), libchipcard2 (built from source), openhbci
(from SuSE), a bunch of XML parsing and internationalization libraries

In a new gnucash file I was able to import my initial checking OFX file
from my bank (about 6 months old, containing about 3 months worth of
transactions) to build my accounts. I import the savings OFX file from
my bank, and it imports (asking me for an account). The next OFX import
of my checking, tries to import into savings account. (I think I have
this issue identified, and a work around identified. My bank identifies
all account types (checking, savings, money market) under the same
account number) After, using a bogus account for a savings import, I was
able to import the next checking account OFX export file into my gnucahs
checking account. When I go to import the checking account export OFX,
the Genric Import Transaction Matcher is empty. In the version provided
with SuSE 10, there was an error if I ran gnucash with --debug. But now,
I don't see anything of much value. If I run with --loglevel 6, I see
one message about it discarding transaction due to a duplicate
transaction number, however, it looks like that is on one record, not
the entire import. If I use ofxdump on the checking export OFX files,
after a few unrecognized container entries are noted to standard error,
I get a full listing of the transactions (Note: I get the same standard
error messages on the initial working OFX export file.) I notice that
the entry "Financial institution's ID for this transaction" starts at 1
and increments for each transaction for each OFX dump. Is this number
supposed to be unique for each successive transaction?

Given the effort and additional libraries I needed to find outside the
distribution, I'm not surprised that a newer version has not been
included in SuSE (although I attribute it to laziness, without any other
explanation from them). Can anyone provide some suggestions on how to
address this?

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

ddougan
On Sun, 2005-11-20 at 10:02 -0500, James Roman wrote:

> I've read the previous thread. Unfortunately after finally getting my
> system updated enough to install the gnucash-1.8.12-3.i586.rpm from
> ftp.suse.com/pub/suse/i386/supplementary/GNOME/update_for_10.0/applications I still can not import OFX files from my bank.
>
> So here are the details:
> I upgraded to SuSE 10 after having the same issue with SuSe 9.3.
> Intel Laptop
> Gnucash version: 1.8.12
> libofx version: 0.8.0-3
> Numerous new applications/libraries needed to meet install requirements
> for gnucahs, including aqbanking (built from source rpm), gwenhywfar
> (built from source rpm), libchipcard2 (built from source), openhbci
> (from SuSE), a bunch of XML parsing and internationalization libraries
>
> In a new gnucash file I was able to import my initial checking OFX file
> from my bank (about 6 months old, containing about 3 months worth of
> transactions) to build my accounts. I import the savings OFX file from
> my bank, and it imports (asking me for an account). The next OFX import
> of my checking, tries to import into savings account. (I think I have
> this issue identified, and a work around identified. My bank identifies
> all account types (checking, savings, money market) under the same
> account number) After, using a bogus account for a savings import, I was
> able to import the next checking account OFX export file into my gnucahs
> checking account. When I go to import the checking account export OFX,
> the Genric Import Transaction Matcher is empty. In the version provided
> with SuSE 10, there was an error if I ran gnucash with --debug. But now,
> I don't see anything of much value. If I run with --loglevel 6, I see
> one message about it discarding transaction due to a duplicate
> transaction number, however, it looks like that is on one record, not
> the entire import. If I use ofxdump on the checking export OFX files,
> after a few unrecognized container entries are noted to standard error,
> I get a full listing of the transactions (Note: I get the same standard
> error messages on the initial working OFX export file.) I notice that
> the entry "Financial institution's ID for this transaction" starts at 1
> and increments for each transaction for each OFX dump. Is this number
> supposed to be unique for each successive transaction?
>
> Given the effort and additional libraries I needed to find outside the
> distribution, I'm not surprised that a newer version has not been
> included in SuSE (although I attribute it to laziness, without any other
> explanation from them). Can anyone provide some suggestions on how to
> address this?

The RPMs for SuSE 10.0 have been separated. There is a separate ofx RPM:

sempron:~ # rpm -qa | grep gnucash
gnucash-1.8.12-2
gnucash-ofx-1.8.12-2
gnucash-docs-1.8.5-1


HTH,

Des
--

Des Dougan, Principal
Dougan Consulting Group

Ph: 604-980-2848       Email: des at DouganConsulting dot com    

        www.DouganConsulting.com

Design - Implementation - Support

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

James Roman
In the installation I installed, the OFX support was not separate. (You
are referring to the installation from
ftp://ftp.gwdg.de/pub/linux/misc/suser-crauch/10.0/RPMS/i686/.) On a
whim, I installed that version of Gnucash, and the gnucash-ofx package,
but I still get the same result, no entries in the Generic Import
Transaction Matcher.

On Sun, 2005-11-20 at 13:53 -0800, Des Dougan wrote:

> On Sun, 2005-11-20 at 10:02 -0500, James Roman wrote:
> > I've read the previous thread. Unfortunately after finally getting my
> > system updated enough to install the gnucash-1.8.12-3.i586.rpm from
> > ftp.suse.com/pub/suse/i386/supplementary/GNOME/update_for_10.0/applications I still can not import OFX files from my bank.
> >
> > So here are the details:
> > I upgraded to SuSE 10 after having the same issue with SuSe 9.3.
> > Intel Laptop
> > Gnucash version: 1.8.12
> > libofx version: 0.8.0-3
> > Numerous new applications/libraries needed to meet install requirements
> > for gnucahs, including aqbanking (built from source rpm), gwenhywfar
> > (built from source rpm), libchipcard2 (built from source), openhbci
> > (from SuSE), a bunch of XML parsing and internationalization libraries
> >
> > In a new gnucash file I was able to import my initial checking OFX file
> > from my bank (about 6 months old, containing about 3 months worth of
> > transactions) to build my accounts. I import the savings OFX file from
> > my bank, and it imports (asking me for an account). The next OFX import
> > of my checking, tries to import into savings account. (I think I have
> > this issue identified, and a work around identified. My bank identifies
> > all account types (checking, savings, money market) under the same
> > account number) After, using a bogus account for a savings import, I was
> > able to import the next checking account OFX export file into my gnucahs
> > checking account. When I go to import the checking account export OFX,
> > the Genric Import Transaction Matcher is empty. In the version provided
> > with SuSE 10, there was an error if I ran gnucash with --debug. But now,
> > I don't see anything of much value. If I run with --loglevel 6, I see
> > one message about it discarding transaction due to a duplicate
> > transaction number, however, it looks like that is on one record, not
> > the entire import. If I use ofxdump on the checking export OFX files,
> > after a few unrecognized container entries are noted to standard error,
> > I get a full listing of the transactions (Note: I get the same standard
> > error messages on the initial working OFX export file.) I notice that
> > the entry "Financial institution's ID for this transaction" starts at 1
> > and increments for each transaction for each OFX dump. Is this number
> > supposed to be unique for each successive transaction?
> >
> > Given the effort and additional libraries I needed to find outside the
> > distribution, I'm not surprised that a newer version has not been
> > included in SuSE (although I attribute it to laziness, without any other
> > explanation from them). Can anyone provide some suggestions on how to
> > address this?
>
> The RPMs for SuSE 10.0 have been separated. There is a separate ofx RPM:
>
> sempron:~ # rpm -qa | grep gnucash
> gnucash-1.8.12-2
> gnucash-ofx-1.8.12-2
> gnucash-docs-1.8.5-1
>
>
> HTH,
>
> Des

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

ddougan
On Sun, 2005-11-20 at 22:11 -0500, James Roman wrote:
> In the installation I installed, the OFX support was not separate. (You
> are referring to the installation from
> ftp://ftp.gwdg.de/pub/linux/misc/suser-crauch/10.0/RPMS/i686/.) On a
> whim, I installed that version of Gnucash, and the gnucash-ofx package,
> but I still get the same result, no entries in the Generic Import
> Transaction Matcher.

Hmmm. If you've installed Christian Rauch's version, then it definitely
works, as I've confirmed that for myself recently. The only reason that
it shows no entries is when there is nothing to match (again something I
was able to confirm recently when I was testing the version - I was very
cautious after installing it, as I've had some real problems with OFX
support). Are you sure that there should be matches at the moment?

Des
--

Des Dougan, Principal
Dougan Consulting Group

Ph: 604-980-2848       Email: des at DouganConsulting dot com    

        www.DouganConsulting.com

Design - Implementation - Support

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

Brian Dolbec
On Sun, 2005-20-11 at 19:16 -0800, Des Dougan wrote:

> On Sun, 2005-11-20 at 22:11 -0500, James Roman wrote:
> > In the installation I installed, the OFX support was not separate. (You
> > are referring to the installation from
> > ftp://ftp.gwdg.de/pub/linux/misc/suser-crauch/10.0/RPMS/i686/.) On a
> > whim, I installed that version of Gnucash, and the gnucash-ofx package,
> > but I still get the same result, no entries in the Generic Import
> > Transaction Matcher.
>
> Hmmm. If you've installed Christian Rauch's version, then it definitely
> works, as I've confirmed that for myself recently. The only reason that
> it shows no entries is when there is nothing to match (again something I
> was able to confirm recently when I was testing the version - I was very
> cautious after installing it, as I've had some real problems with OFX
> support). Are you sure that there should be matches at the moment?
>
> Des

I remember that someone else had problems importing stuff.  It turned
out the date identifier did not match a gnucash known type.  It
prevented everything from working to find matches.  Check the import
file in an editor to see if it looks alright.  Check the archives for
the thread if that might be your problem as well.  I believe he did a
search & replace on the identifier for it to work.

--
Brian Dolbec <[hidden email]>

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

James Roman
I think that mine is a little bit more difficult to fix. The date
identifier worked for the initial import, but none after. I did an
export from another bank account for my business. After viewing the 2nd
bank export, it became clear what the problem was. My 1st bank exports
do not have valid <FITID> entries. These look like they should be unique
numbers (and fairly long at that). In the export files that I have, the
numbers correspond to the number of transactions in the export file
(I.E. transaction 1 has FITID=1, etc.)

Since I don't expect my bank to fix this problem soon, I'll have to come
up with an way of modifying these numbers to make them unique. Ideally,
I should probably come up with a way that if I exported overlapping date
ranges, it would identify them as the same transaction. Anyone have any
suggestions how I might accomplish this?

On Sun, 2005-11-20 at 23:50 -0800, Brian Dolbec wrote:

>
> I remember that someone else had problems importing stuff.  It turned
> out the date identifier did not match a gnucash known type.  It
> prevented everything from working to find matches.  Check the import
> file in an editor to see if it looks alright.  Check the archives for
> the thread if that might be your problem as well.  I believe he did a
> search & replace on the identifier for it to work.
>

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

Andrea Carpani
Il giorno lun, 21-11-2005 alle 18:16 -0500, James Roman ha scritto:

> Since I don't expect my bank to fix this problem soon, I'll have to come
> up with an way of modifying these numbers to make them unique. Ideally,
> I should probably come up with a way that if I exported overlapping date
> ranges, it would identify them as the same transaction. Anyone have any
> suggestions how I might accomplish this?

I have a similar problem in that my bank uses 0 as FITID in OFX files
for all transactions this results in a single imported transaction.
I'm creating a perl script that changes FITID for each transaction
setting it to a md5 string

$T{FITID} = md5_hex("$T{TRNTYPE}.$T{TRNAMT}.$T{MEMO}");

I yet have to test it as I'm rebuilding gnucash at the moment.

Question: should FITID be numeric?
I'll find out later...

--

[hidden email]
http://www.carpani.net
Andrea Carpani
.a.c.

--
Andrea Carpani <[hidden email]>

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

David Reiser

On Nov 22, 2005, at 5:34 AM, Andrea Carpani wrote:

> Il giorno lun, 21-11-2005 alle 18:16 -0500, James Roman ha scritto:
>
>> Since I don't expect my bank to fix this problem soon, I'll have  
>> to come
>> up with an way of modifying these numbers to make them unique.  
>> Ideally,
>> I should probably come up with a way that if I exported  
>> overlapping date
>> ranges, it would identify them as the same transaction. Anyone  
>> have any
>> suggestions how I might accomplish this?
>
> I have a similar problem in that my bank uses 0 as FITID in OFX files
> for all transactions this results in a single imported transaction.
> I'm creating a perl script that changes FITID for each transaction
> setting it to a md5 string
>
> $T{FITID} = md5_hex("$T{TRNTYPE}.$T{TRNAMT}.$T{MEMO}");

My experience with OFX imports is that MEMO will be the same for at  
least the same merchant, and sometimes for different merchants.  
TRNTYPE is also not likely to vary much. I would suggest putting the  
transaction date in your md5 input as means of ensuring you maintain  
uniqueness.


>
> I yet have to test it as I'm rebuilding gnucash at the moment.
>
> Question: should FITID be numeric?

Doesn't have to be. One of my credit card issuers uses: full credit  
card number+8 digit date+ 11 digits for amount (where the minus sign  
for a purchase appears mid FTID) + 6 digits for a sequence number in  
the download.

A couple other organizations just use a 9 or 10 digit number

> I'll find out later...
>
> --
>
> [hidden email]
> http://www.carpani.net
> Andrea Carpani
> .a.c.
>
> --
> Andrea Carpani <[hidden email]>
>

--
David Reiser
[hidden email]

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

James Roman
In reply to this post by Andrea Carpani
Thanks for the response. I am leaning towards using perl as well. I was
planning on changing FITID to DTPOSTED+daily sequence number. I think
your suggestion has merit, though. I didn't like the idea of being at
the mercy the export program with the sequence number. I think that the
only required fields (other than the bogus FITID) are DTPOSTED, TRNAMT
and TRNTYPE. Using these three fields seems like it might be the most
complete.

I'm going to think about this for a day or two. All my transactions are
listed at the same date stamp for DTPOSTED. I want to consider some sort
of odd situation, like whe you first sign up for automatic deposits,
when they deposit one dollar, and then take it back immediately.
Additionally, I wonder what would happen if I had multiple accounts with
a bank.

Let me know how your solution comes along.

Below is the section from the OFX Specification of FITIDs. They do not
need to be numeric,but should be less than 255 characters (I'd avoid <,
> or / as they might be interpreted as a field.

3.2.3 Financial Institution Transaction ID <FITID>
Format: A-255
An FI (or its Service Provider) assigns an <FITID> to uniquely identify
a financial transaction that can
appear in an account statement. Its primary purpose is to allow a client
to detect duplicate responses. Open
Financial Exchange intends <FITID> for use in statement download
applications, where every transaction
(not just those that are client-originated or server-originated)
requires a unique ID.
An <FITID> also uniquely identifies the closing statement in <CLOSINGRS>
and <CCCLOSINGRS>.
Again, the OFX client should detect repeated closing statements
(duplicate downloads) using these
identifiers.
FITIDs must be unique within the scope of an account but need not be
sequential or even increasing.
Clients should be aware that FITIDs are not unique across FIs. If a
client performs the same type of request
within the same scope at two different FIs, clients will need to use FI
+ <ACCTID> + <FITID> as a
globally unique key in a client database. That is, the <FITID> value
must be unique within the account and
Financial Institution (independent of the service provider).
Note: Although the specification allows FITIDs of up to 255 characters,
client performance
may significantly improve if servers use fewer characters. It is
recommended that servers use
32 characters or fewer.
For example: The two spawned payments mentioned in section 3.2.1 are
processed and later downloaded
in a <STMTRS>. The first payment’s <STMTTRN> would list
<SRVRTID>8765:4321,
<RECSRVRTID>1234:5678, and <FITID>9999:8888:7777. The second payment
would be described in a
<STMTTRN> containing <SRVRTID>8765:4322, <RECSRVRTID>1234:5678, and
<FITID>6666:5555:4444.
Usage: Bank statement download, investment statement download
Elements of this type: <CORRECTFITID>, <FITID>, <RELFITID>, and
<REVERSALFITID>

On Tue, 2005-11-22 at 11:34 +0100, Andrea Carpani wrote:

> Il giorno lun, 21-11-2005 alle 18:16 -0500, James Roman ha scritto:
>
> > Since I don't expect my bank to fix this problem soon, I'll have to come
> > up with an way of modifying these numbers to make them unique. Ideally,
> > I should probably come up with a way that if I exported overlapping date
> > ranges, it would identify them as the same transaction. Anyone have any
> > suggestions how I might accomplish this?
>
> I have a similar problem in that my bank uses 0 as FITID in OFX files
> for all transactions this results in a single imported transaction.
> I'm creating a perl script that changes FITID for each transaction
> setting it to a md5 string
>
> $T{FITID} = md5_hex("$T{TRNTYPE}.$T{TRNAMT}.$T{MEMO}");
>
> I yet have to test it as I'm rebuilding gnucash at the moment.
>
> Question: should FITID be numeric?
> I'll find out later...
>
> --
>
> [hidden email]
> http://www.carpani.net
> Andrea Carpani
> .a.c.
>

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

Derek Atkins
Perhaps the importer should just be changed to allow a FITID of '0'
to mean "no id" and ignore duplicates?

-derek

James Roman <[hidden email]> writes:

> Thanks for the response. I am leaning towards using perl as well. I was
> planning on changing FITID to DTPOSTED+daily sequence number. I think
> your suggestion has merit, though. I didn't like the idea of being at
> the mercy the export program with the sequence number. I think that the
> only required fields (other than the bogus FITID) are DTPOSTED, TRNAMT
> and TRNTYPE. Using these three fields seems like it might be the most
> complete.
>
> I'm going to think about this for a day or two. All my transactions are
> listed at the same date stamp for DTPOSTED. I want to consider some sort
> of odd situation, like whe you first sign up for automatic deposits,
> when they deposit one dollar, and then take it back immediately.
> Additionally, I wonder what would happen if I had multiple accounts with
> a bank.
>
> Let me know how your solution comes along.
>
> Below is the section from the OFX Specification of FITIDs. They do not
> need to be numeric,but should be less than 255 characters (I'd avoid <,
>> or / as they might be interpreted as a field.
>
> 3.2.3 Financial Institution Transaction ID <FITID>
> Format: A-255
> An FI (or its Service Provider) assigns an <FITID> to uniquely identify
> a financial transaction that can
> appear in an account statement. Its primary purpose is to allow a client
> to detect duplicate responses. Open
> Financial Exchange intends <FITID> for use in statement download
> applications, where every transaction
> (not just those that are client-originated or server-originated)
> requires a unique ID.
> An <FITID> also uniquely identifies the closing statement in <CLOSINGRS>
> and <CCCLOSINGRS>.
> Again, the OFX client should detect repeated closing statements
> (duplicate downloads) using these
> identifiers.
> FITIDs must be unique within the scope of an account but need not be
> sequential or even increasing.
> Clients should be aware that FITIDs are not unique across FIs. If a
> client performs the same type of request
> within the same scope at two different FIs, clients will need to use FI
> + <ACCTID> + <FITID> as a
> globally unique key in a client database. That is, the <FITID> value
> must be unique within the account and
> Financial Institution (independent of the service provider).
> Note: Although the specification allows FITIDs of up to 255 characters,
> client performance
> may significantly improve if servers use fewer characters. It is
> recommended that servers use
> 32 characters or fewer.
> For example: The two spawned payments mentioned in section 3.2.1 are
> processed and later downloaded
> in a <STMTRS>. The first payment’s <STMTTRN> would list
> <SRVRTID>8765:4321,
> <RECSRVRTID>1234:5678, and <FITID>9999:8888:7777. The second payment
> would be described in a
> <STMTTRN> containing <SRVRTID>8765:4322, <RECSRVRTID>1234:5678, and
> <FITID>6666:5555:4444.
> Usage: Bank statement download, investment statement download
> Elements of this type: <CORRECTFITID>, <FITID>, <RELFITID>, and
> <REVERSALFITID>
>
> On Tue, 2005-11-22 at 11:34 +0100, Andrea Carpani wrote:
>> Il giorno lun, 21-11-2005 alle 18:16 -0500, James Roman ha scritto:
>>
>> > Since I don't expect my bank to fix this problem soon, I'll have to come
>> > up with an way of modifying these numbers to make them unique. Ideally,
>> > I should probably come up with a way that if I exported overlapping date
>> > ranges, it would identify them as the same transaction. Anyone have any
>> > suggestions how I might accomplish this?
>>
>> I have a similar problem in that my bank uses 0 as FITID in OFX files
>> for all transactions this results in a single imported transaction.
>> I'm creating a perl script that changes FITID for each transaction
>> setting it to a md5 string
>>
>> $T{FITID} = md5_hex("$T{TRNTYPE}.$T{TRNAMT}.$T{MEMO}");
>>
>> I yet have to test it as I'm rebuilding gnucash at the moment.
>>
>> Question: should FITID be numeric?
>> I'll find out later...
>>
>> --
>>
>> [hidden email]
>> http://www.carpani.net
>> Andrea Carpani
>> .a.c.
>>
> _______________________________________________
> gnucash-user mailing list
> [hidden email]
> https://lists.gnucash.org/mailman/listinfo/gnucash-user

--
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       [hidden email]                        PGP key available

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

Andrea Carpani
In reply to this post by James Roman
Il giorno mar, 22-11-2005 alle 22:53 -0500, James Roman ha scritto:

> Thanks for the response. I am leaning towards using perl as well. I was
> planning on changing FITID to DTPOSTED+daily sequence number.

The problem with daily sequence number is that you could have exported
to OFX only some transactions through a filter and calculating the value
could be difficult.

DTPOSTED form me is:

<DTPOSTED>20050913000000

and stays the same for the whole day :(

On the other hand MEMO has some uniqueness: transaction endpoint
coordinates and, sometimes, HH:MM of transaction.

> I think
> your suggestion has merit, though. I didn't like the idea of being at
> the mercy the export program with the sequence number.

True, it doesn't sound very safe.

> I think that the
> only required fields (other than the bogus FITID) are DTPOSTED, TRNAMT
> and TRNTYPE. Using these three fields seems like it might be the most
> complete.

With the exports I get from my bank MEMO is quite good, while TRNTYPE,
as Reiser noted, doesn't change very much.

> I'm going to think about this for a day or two. All my transactions are
> listed at the same date stamp for DTPOSTED. I want to consider some sort
> of odd situation, like whe you first sign up for automatic deposits,
> when they deposit one dollar, and then take it back immediately.
> Additionally, I wonder what would happen if I had multiple accounts with
> a bank.
>
> Let me know how your solution comes along.

I'm using my perl script to add FITID calculated as:

$T{FITID} = md5_hex("$T{DTPOSTED}.$T{TRNAMT}.$T{MEMO}");

I still have to understand the basics of gnucash, so I still have to
know if it works well or not. At least I can import my data :)

--
Andrea Carpani <[hidden email]>

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

James Roman
In reply to this post by Derek Atkins
Unfortunately, the OFX files from my bank number each FITID from 1 to X.
The next month they start the sequence over again (numbering the
transactions in the OFX file).

In my instance I would prefer falling back to some alternate numbering
schema, where an actual duplicate transaction could be identified.

On Mon, 2005-11-28 at 09:34 -0500, Derek Atkins wrote:

> Perhaps the importer should just be changed to allow a FITID of '0'
> to mean "no id" and ignore duplicates?
>
> -derek
>
> James Roman <[hidden email]> writes:
>
> > Thanks for the response. I am leaning towards using perl as well. I was
> > planning on changing FITID to DTPOSTED+daily sequence number. I think
> > your suggestion has merit, though. I didn't like the idea of being at
> > the mercy the export program with the sequence number. I think that the
> > only required fields (other than the bogus FITID) are DTPOSTED, TRNAMT
> > and TRNTYPE. Using these three fields seems like it might be the most
> > complete.
> >
> > I'm going to think about this for a day or two. All my transactions are
> > listed at the same date stamp for DTPOSTED. I want to consider some sort
> > of odd situation, like whe you first sign up for automatic deposits,
> > when they deposit one dollar, and then take it back immediately.
> > Additionally, I wonder what would happen if I had multiple accounts with
> > a bank.
> >
> > Let me know how your solution comes along.
> >
> > Below is the section from the OFX Specification of FITIDs. They do not
> > need to be numeric,but should be less than 255 characters (I'd avoid <,
> >> or / as they might be interpreted as a field.
> >
> > 3.2.3 Financial Institution Transaction ID <FITID>
> > Format: A-255
> > An FI (or its Service Provider) assigns an <FITID> to uniquely identify
> > a financial transaction that can
> > appear in an account statement. Its primary purpose is to allow a client
> > to detect duplicate responses. Open
> > Financial Exchange intends <FITID> for use in statement download
> > applications, where every transaction
> > (not just those that are client-originated or server-originated)
> > requires a unique ID.
> > An <FITID> also uniquely identifies the closing statement in <CLOSINGRS>
> > and <CCCLOSINGRS>.
> > Again, the OFX client should detect repeated closing statements
> > (duplicate downloads) using these
> > identifiers.
> > FITIDs must be unique within the scope of an account but need not be
> > sequential or even increasing.
> > Clients should be aware that FITIDs are not unique across FIs. If a
> > client performs the same type of request
> > within the same scope at two different FIs, clients will need to use FI
> > + <ACCTID> + <FITID> as a
> > globally unique key in a client database. That is, the <FITID> value
> > must be unique within the account and
> > Financial Institution (independent of the service provider).
> > Note: Although the specification allows FITIDs of up to 255 characters,
> > client performance
> > may significantly improve if servers use fewer characters. It is
> > recommended that servers use
> > 32 characters or fewer.
> > For example: The two spawned payments mentioned in section 3.2.1 are
> > processed and later downloaded
> > in a <STMTRS>. The first payment’s <STMTTRN> would list
> > <SRVRTID>8765:4321,
> > <RECSRVRTID>1234:5678, and <FITID>9999:8888:7777. The second payment
> > would be described in a
> > <STMTTRN> containing <SRVRTID>8765:4322, <RECSRVRTID>1234:5678, and
> > <FITID>6666:5555:4444.
> > Usage: Bank statement download, investment statement download
> > Elements of this type: <CORRECTFITID>, <FITID>, <RELFITID>, and
> > <REVERSALFITID>
> >
> > On Tue, 2005-11-22 at 11:34 +0100, Andrea Carpani wrote:
> >> Il giorno lun, 21-11-2005 alle 18:16 -0500, James Roman ha scritto:
> >>
> >> > Since I don't expect my bank to fix this problem soon, I'll have to come
> >> > up with an way of modifying these numbers to make them unique. Ideally,
> >> > I should probably come up with a way that if I exported overlapping date
> >> > ranges, it would identify them as the same transaction. Anyone have any
> >> > suggestions how I might accomplish this?
> >>
> >> I have a similar problem in that my bank uses 0 as FITID in OFX files
> >> for all transactions this results in a single imported transaction.
> >> I'm creating a perl script that changes FITID for each transaction
> >> setting it to a md5 string
> >>
> >> $T{FITID} = md5_hex("$T{TRNTYPE}.$T{TRNAMT}.$T{MEMO}");
> >>
> >> I yet have to test it as I'm rebuilding gnucash at the moment.
> >>
> >> Question: should FITID be numeric?
> >> I'll find out later...
> >>
> >> --
> >>
> >> [hidden email]
> >> http://www.carpani.net
> >> Andrea Carpani
> >> .a.c.
> >>
> > _______________________________________________
> > gnucash-user mailing list
> > [hidden email]
> > https://lists.gnucash.org/mailman/listinfo/gnucash-user
>

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

James Roman
In reply to this post by Andrea Carpani
On Mon, 2005-11-28 at 17:31 +0100, Andrea Carpani wrote:

> Il giorno mar, 22-11-2005 alle 22:53 -0500, James Roman ha scritto:
>
> > Thanks for the response. I am leaning towards using perl as well. I was
> > planning on changing FITID to DTPOSTED+daily sequence number.
>
> The problem with daily sequence number is that you could have exported
> to OFX only some transactions through a filter and calculating the value
> could be difficult.
>
> DTPOSTED form me is:
>
> <DTPOSTED>20050913000000
>
> and stays the same for the whole day :(
>
> On the other hand MEMO has some uniqueness: transaction endpoint
> coordinates and, sometimes, HH:MM of transaction.
>
> > I think
> > your suggestion has merit, though. I didn't like the idea of being at
> > the mercy the export program with the sequence number.
>
> True, it doesn't sound very safe.
>
> > I think that the
> > only required fields (other than the bogus FITID) are DTPOSTED, TRNAMT
> > and TRNTYPE. Using these three fields seems like it might be the most
> > complete.
>
> With the exports I get from my bank MEMO is quite good, while TRNTYPE,
> as Reiser noted, doesn't change very much.
>
> > I'm going to think about this for a day or two. All my transactions are
> > listed at the same date stamp for DTPOSTED. I want to consider some sort
> > of odd situation, like whe you first sign up for automatic deposits,
> > when they deposit one dollar, and then take it back immediately.
> > Additionally, I wonder what would happen if I had multiple accounts with
> > a bank.
> >
> > Let me know how your solution comes along.
>
> I'm using my perl script to add FITID calculated as:
>
> $T{FITID} = md5_hex("$T{DTPOSTED}.$T{TRNAMT}.$T{MEMO}");
>
> I still have to understand the basics of gnucash, so I still have to
> know if it works well or not. At least I can import my data :)
>
My only reservation with the Memo field is that it is not required. I
have a handful of transactions that do not include a Memo. I would
suspect that 99.99% of the time, you would still get a unique number.
(Just avoid open air markets in Stockholm.)

_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

Derek Atkins
In reply to this post by James Roman

Oh.  Well THAT is illegal.  You should complain to your bank about
that.  Tell them quicken fails to import the transactions, and
that when you asked Intuit about they they told you to complain to
the bank and that the bank was violating the OFX Specification.

-derek

Quoting James Roman <[hidden email]>:

> Unfortunately, the OFX files from my bank number each FITID from 1 to X.
> The next month they start the sequence over again (numbering the
> transactions in the OFX file).
>
> In my instance I would prefer falling back to some alternate numbering
> schema, where an actual duplicate transaction could be identified.
>
> On Mon, 2005-11-28 at 09:34 -0500, Derek Atkins wrote:
>> Perhaps the importer should just be changed to allow a FITID of '0'
>> to mean "no id" and ignore duplicates?
>>
>> -derek
>>
>> James Roman <[hidden email]> writes:
>>
>> > Thanks for the response. I am leaning towards using perl as well. I was
>> > planning on changing FITID to DTPOSTED+daily sequence number. I think
>> > your suggestion has merit, though. I didn't like the idea of being at
>> > the mercy the export program with the sequence number. I think that the
>> > only required fields (other than the bogus FITID) are DTPOSTED, TRNAMT
>> > and TRNTYPE. Using these three fields seems like it might be the most
>> > complete.
>> >
>> > I'm going to think about this for a day or two. All my transactions are
>> > listed at the same date stamp for DTPOSTED. I want to consider some sort
>> > of odd situation, like whe you first sign up for automatic deposits,
>> > when they deposit one dollar, and then take it back immediately.
>> > Additionally, I wonder what would happen if I had multiple accounts with
>> > a bank.
>> >
>> > Let me know how your solution comes along.
>> >
>> > Below is the section from the OFX Specification of FITIDs. They do not
>> > need to be numeric,but should be less than 255 characters (I'd avoid <,
>> >> or / as they might be interpreted as a field.
>> >
>> > 3.2.3 Financial Institution Transaction ID <FITID>
>> > Format: A-255
>> > An FI (or its Service Provider) assigns an <FITID> to uniquely identify
>> > a financial transaction that can
>> > appear in an account statement. Its primary purpose is to allow a client
>> > to detect duplicate responses. Open
>> > Financial Exchange intends <FITID> for use in statement download
>> > applications, where every transaction
>> > (not just those that are client-originated or server-originated)
>> > requires a unique ID.
>> > An <FITID> also uniquely identifies the closing statement in <CLOSINGRS>
>> > and <CCCLOSINGRS>.
>> > Again, the OFX client should detect repeated closing statements
>> > (duplicate downloads) using these
>> > identifiers.
>> > FITIDs must be unique within the scope of an account but need not be
>> > sequential or even increasing.
>> > Clients should be aware that FITIDs are not unique across FIs. If a
>> > client performs the same type of request
>> > within the same scope at two different FIs, clients will need to use FI
>> > + <ACCTID> + <FITID> as a
>> > globally unique key in a client database. That is, the <FITID> value
>> > must be unique within the account and
>> > Financial Institution (independent of the service provider).
>> > Note: Although the specification allows FITIDs of up to 255 characters,
>> > client performance
>> > may significantly improve if servers use fewer characters. It is
>> > recommended that servers use
>> > 32 characters or fewer.
>> > For example: The two spawned payments mentioned in section 3.2.1 are
>> > processed and later downloaded
>> > in a <STMTRS>. The first payment’s <STMTTRN> would list
>> > <SRVRTID>8765:4321,
>> > <RECSRVRTID>1234:5678, and <FITID>9999:8888:7777. The second payment
>> > would be described in a
>> > <STMTTRN> containing <SRVRTID>8765:4322, <RECSRVRTID>1234:5678, and
>> > <FITID>6666:5555:4444.
>> > Usage: Bank statement download, investment statement download
>> > Elements of this type: <CORRECTFITID>, <FITID>, <RELFITID>, and
>> > <REVERSALFITID>
>> >
>> > On Tue, 2005-11-22 at 11:34 +0100, Andrea Carpani wrote:
>> >> Il giorno lun, 21-11-2005 alle 18:16 -0500, James Roman ha scritto:
>> >>
>> >> > Since I don't expect my bank to fix this problem soon, I'll
>> have to come
>> >> > up with an way of modifying these numbers to make them unique. Ideally,
>> >> > I should probably come up with a way that if I exported
>> overlapping date
>> >> > ranges, it would identify them as the same transaction. Anyone have any
>> >> > suggestions how I might accomplish this?
>> >>
>> >> I have a similar problem in that my bank uses 0 as FITID in OFX files
>> >> for all transactions this results in a single imported transaction.
>> >> I'm creating a perl script that changes FITID for each transaction
>> >> setting it to a md5 string
>> >>
>> >> $T{FITID} = md5_hex("$T{TRNTYPE}.$T{TRNAMT}.$T{MEMO}");
>> >>
>> >> I yet have to test it as I'm rebuilding gnucash at the moment.
>> >>
>> >> Question: should FITID be numeric?
>> >> I'll find out later...
>> >>
>> >> --
>> >>
>> >> [hidden email]
>> >> http://www.carpani.net
>> >> Andrea Carpani
>> >> .a.c.
>> >>
>> > _______________________________________________
>> > gnucash-user mailing list
>> > [hidden email]
>> > https://lists.gnucash.org/mailman/listinfo/gnucash-user
>>
>



--
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       [hidden email]                        PGP key available


_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user
Reply | Threaded
Open this post in threaded view
|

Re: Generic Import Transaction Matcher empty - revisited

Andrew Sackville-West


Derek Atkins wrote:
>
> Oh.  Well THAT is illegal.  You should complain to your bank about
> that.  Tell them quicken fails to import the transactions, and
> that when you asked Intuit about they they told you to complain to
> the bank and that the bank was violating the OFX Specification.

oh man I love it....

:)

A

>
> -derek
>
> Quoting James Roman <[hidden email]>:
>
>> Unfortunately, the OFX files from my bank number each FITID from 1 to X.
>> The next month they start the sequence over again (numbering the
>> transactions in the OFX file).
>>
>> In my instance I would prefer falling back to some alternate numbering
>> schema, where an actual duplicate transaction could be identified.
>>
>> On Mon, 2005-11-28 at 09:34 -0500, Derek Atkins wrote:
>>
>>> Perhaps the importer should just be changed to allow a FITID of '0'
>>> to mean "no id" and ignore duplicates?
>>>
>>> -derek
>>>
>>> James Roman <[hidden email]> writes:
>>>
>>> > Thanks for the response. I am leaning towards using perl as well. I
>>> was
>>> > planning on changing FITID to DTPOSTED+daily sequence number. I think
>>> > your suggestion has merit, though. I didn't like the idea of being at
>>> > the mercy the export program with the sequence number. I think that
>>> the
>>> > only required fields (other than the bogus FITID) are DTPOSTED, TRNAMT
>>> > and TRNTYPE. Using these three fields seems like it might be the most
>>> > complete.
>>> >
>>> > I'm going to think about this for a day or two. All my transactions
>>> are
>>> > listed at the same date stamp for DTPOSTED. I want to consider some
>>> sort
>>> > of odd situation, like whe you first sign up for automatic deposits,
>>> > when they deposit one dollar, and then take it back immediately.
>>> > Additionally, I wonder what would happen if I had multiple accounts
>>> with
>>> > a bank.
>>> >
>>> > Let me know how your solution comes along.
>>> >
>>> > Below is the section from the OFX Specification of FITIDs. They do not
>>> > need to be numeric,but should be less than 255 characters (I'd
>>> avoid <,
>>> >> or / as they might be interpreted as a field.
>>> >
>>> > 3.2.3 Financial Institution Transaction ID <FITID>
>>> > Format: A-255
>>> > An FI (or its Service Provider) assigns an <FITID> to uniquely
>>> identify
>>> > a financial transaction that can
>>> > appear in an account statement. Its primary purpose is to allow a
>>> client
>>> > to detect duplicate responses. Open
>>> > Financial Exchange intends <FITID> for use in statement download
>>> > applications, where every transaction
>>> > (not just those that are client-originated or server-originated)
>>> > requires a unique ID.
>>> > An <FITID> also uniquely identifies the closing statement in
>>> <CLOSINGRS>
>>> > and <CCCLOSINGRS>.
>>> > Again, the OFX client should detect repeated closing statements
>>> > (duplicate downloads) using these
>>> > identifiers.
>>> > FITIDs must be unique within the scope of an account but need not be
>>> > sequential or even increasing.
>>> > Clients should be aware that FITIDs are not unique across FIs. If a
>>> > client performs the same type of request
>>> > within the same scope at two different FIs, clients will need to
>>> use FI
>>> > + <ACCTID> + <FITID> as a
>>> > globally unique key in a client database. That is, the <FITID> value
>>> > must be unique within the account and
>>> > Financial Institution (independent of the service provider).
>>> > Note: Although the specification allows FITIDs of up to 255
>>> characters,
>>> > client performance
>>> > may significantly improve if servers use fewer characters. It is
>>> > recommended that servers use
>>> > 32 characters or fewer.
>>> > For example: The two spawned payments mentioned in section 3.2.1 are
>>> > processed and later downloaded
>>> > in a <STMTRS>. The first payment’s <STMTTRN> would list
>>> > <SRVRTID>8765:4321,
>>> > <RECSRVRTID>1234:5678, and <FITID>9999:8888:7777. The second payment
>>> > would be described in a
>>> > <STMTTRN> containing <SRVRTID>8765:4322, <RECSRVRTID>1234:5678, and
>>> > <FITID>6666:5555:4444.
>>> > Usage: Bank statement download, investment statement download
>>> > Elements of this type: <CORRECTFITID>, <FITID>, <RELFITID>, and
>>> > <REVERSALFITID>
>>> >
>>> > On Tue, 2005-11-22 at 11:34 +0100, Andrea Carpani wrote:
>>> >> Il giorno lun, 21-11-2005 alle 18:16 -0500, James Roman ha scritto:
>>> >>
>>> >> > Since I don't expect my bank to fix this problem soon, I'll have
>>> to come
>>> >> > up with an way of modifying these numbers to make them unique.
>>> Ideally,
>>> >> > I should probably come up with a way that if I exported
>>> overlapping date
>>> >> > ranges, it would identify them as the same transaction. Anyone
>>> have any
>>> >> > suggestions how I might accomplish this?
>>> >>
>>> >> I have a similar problem in that my bank uses 0 as FITID in OFX files
>>> >> for all transactions this results in a single imported transaction.
>>> >> I'm creating a perl script that changes FITID for each transaction
>>> >> setting it to a md5 string
>>> >>
>>> >> $T{FITID} = md5_hex("$T{TRNTYPE}.$T{TRNAMT}.$T{MEMO}");
>>> >>
>>> >> I yet have to test it as I'm rebuilding gnucash at the moment.
>>> >>
>>> >> Question: should FITID be numeric?
>>> >> I'll find out later...
>>> >>
>>> >> --
>>> >>
>>> >> [hidden email]
>>> >> http://www.carpani.net
>>> >> Andrea Carpani
>>> >> .a.c.
>>> >>
>>> > _______________________________________________
>>> > gnucash-user mailing list
>>> > [hidden email]
>>> > https://lists.gnucash.org/mailman/listinfo/gnucash-user
>>>
>>
>
>
>
_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user