Skip finding duplicate transactions on QIF import

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Skip finding duplicate transactions on QIF import

Jeff Carneal-2

Ok, so I doubt anyone cares but me, but you never know...

I have several largish hand-crafted qif files that I will be  
importing on a monthly basis and while doing that for the first time  
today I noticed that the finding duplicate transaction import code  
was taking forever on my spiffy new intel mac.  To the tune of 5-10  
minutes, that is.

Since I know there are no (or very few duplicates), I wanted to  
simply disable the duplicate transaction checking code.  I finally  
found it in qif-merge-groups.scm and made a very trivial change to  
the gnc:group-find-duplicates function.  Please note this is a much  
of a hack as it could possibly be, and I'm aware of that.  But it's  
going to save me time and I figured there was a chance someone else  
wanted this change as well.

I suppose if enough people were interested I'd code in a preference  
for "finding duplicates" where you could switch this on and off, but  
I doubt that will be the case.  Let me know if I'm mistaken.

Without further ado, here's the function.  Simply replace all of the  
function  gnc:group-find-duplicates in your qif-merge-groups.scm with  
this and you're good to go.

(define (gnc:group-find-duplicates old-group new-group window)
   ;; get all the transactions in the new group, then iterate over them
   ;; trying to find matches in the new group.  If there are matches,
   ;; push the matches onto a list.
   (let* ((new-xtns (gnc:group-get-transactions new-group))
    (progress-dialog #f)
    (work-to-do (length new-xtns))
    (work-done 0)
          (matches '()))
     ;; get rid of the progress dialog
     (if progress-dialog
   (gnc:progress-dialog-destroy progress-dialog))

     ;; return the matches
     matches))

Jeff

PS - No, I don't know scheme so I can't help you with anything else  
scheme-related. Hell, this probably isn't right either but it works :)
_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user
-----
Please remember to CC this list on all your replies.
You can do this by using Reply-To-List or Reply-All.
Reply | Threaded
Open this post in threaded view
|

Re: Skip finding duplicate transactions on QIF import

Derek Atkins
Um, I highly recommend you NOT make this change.  It completely
bypasses duplicate checking in all cases, which is far from correct.

A better fix would be a change to the query/algorithm to make
it faster.

-derek

Jeff Carneal <[hidden email]> writes:

> Ok, so I doubt anyone cares but me, but you never know...
>
> I have several largish hand-crafted qif files that I will be  
> importing on a monthly basis and while doing that for the first time  
> today I noticed that the finding duplicate transaction import code  
> was taking forever on my spiffy new intel mac.  To the tune of 5-10  
> minutes, that is.
>
> Since I know there are no (or very few duplicates), I wanted to  
> simply disable the duplicate transaction checking code.  I finally  
> found it in qif-merge-groups.scm and made a very trivial change to  
> the gnc:group-find-duplicates function.  Please note this is a much  
> of a hack as it could possibly be, and I'm aware of that.  But it's  
> going to save me time and I figured there was a chance someone else  
> wanted this change as well.
>
> I suppose if enough people were interested I'd code in a preference  
> for "finding duplicates" where you could switch this on and off, but  
> I doubt that will be the case.  Let me know if I'm mistaken.
>
> Without further ado, here's the function.  Simply replace all of the  
> function  gnc:group-find-duplicates in your qif-merge-groups.scm with  
> this and you're good to go.
>
> (define (gnc:group-find-duplicates old-group new-group window)
>    ;; get all the transactions in the new group, then iterate over them
>    ;; trying to find matches in the new group.  If there are matches,
>    ;; push the matches onto a list.
>    (let* ((new-xtns (gnc:group-get-transactions new-group))
>     (progress-dialog #f)
>     (work-to-do (length new-xtns))
>     (work-done 0)
>           (matches '()))
>      ;; get rid of the progress dialog
>      (if progress-dialog
>    (gnc:progress-dialog-destroy progress-dialog))
>
>      ;; return the matches
>      matches))
>
> Jeff
>
> PS - No, I don't know scheme so I can't help you with anything else  
> scheme-related. Hell, this probably isn't right either but it works :)
> _______________________________________________
> gnucash-user mailing list
> [hidden email]
> https://lists.gnucash.org/mailman/listinfo/gnucash-user
> -----
> Please remember to CC this list on all your replies.
> You can do this by using Reply-To-List or Reply-All.
>
>

--
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       [hidden email]                        PGP key available
_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user
-----
Please remember to CC this list on all your replies.
You can do this by using Reply-To-List or Reply-All.
Reply | Threaded
Open this post in threaded view
|

Re: Skip finding duplicate transactions on QIF import

Jeff Carneal-4

On Feb 14, 2007, at 9:25 AM, Derek Atkins wrote:

> Um, I highly recommend you NOT make this change.  It completely
> bypasses duplicate checking in all cases, which is far from correct.

Right, I understand that.  That's why I did it.

>
> A better fix would be a change to the query/algorithm to make
> it faster.

I agree.

I also agree with something I believe you wrote about a wholesale  
rewrite of the import code.  While the scheme / C marriage may have  
served its purpose, I also think it's holding back the project from  
being developed and maintained properly.  Perhaps rewriting the QIF  
importer and/or creating a more generic importer written in C is  
something I could undertake as time permits...

Jeff
_______________________________________________
gnucash-user mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-user
-----
Please remember to CC this list on all your replies.
You can do this by using Reply-To-List or Reply-All.