[GNC-dev] Alphavantage strategies

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[GNC-dev] Alphavantage strategies

GnuCash - Dev mailing list
I was unaware [1] until recently that Alphavantage wanted individuals /
non-subscribers / poor people / whoever doesn't pay / to back off a bit.
  I don't have a problem with this, it is a good free service and it has
been obvious they're struggling.

[1] but not surprised to discover

There have been postings in the .user list (one of them being mine) that
involve editing files in the F:Q install to mitigate this.  Not a good
idea as the next update overwrites any changes, users shouldn't have to
get involved, may not be reading the list, sleep(15) is arbitrary, etc.

Let's consider some simple backoff strategies that gnc could implement,
feel free to add your own.

Wm's suggestion
===============
only collect if the most recent price is more than 1 day (or 12 hours,
etc) old [2].  this sort of fits in with the existing post collection
query, i.e. always store the good prices that have been obtained,
instead ask the user, "do you want to try and fetch the other prices /
rates again".  this provides a natural pause to Alphavantage as gnc
won't be hitting it for prices it just got a second time and would only
be requesting the prices it didn't get last time.

[2] I use this in my own collections (external to gnc) run by cron, some
more obscure currency pairs are only available a few hours a day, I
don't even try if the most recent is less than a day old.

--
Wm













_______________________________________________
gnucash-devel mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Reply | Threaded
Open this post in threaded view
|

Re: [GNC-dev] Alphavantage strategies

Derek Atkins-3
Hi,

Wm via gnucash-devel <[hidden email]> writes:

[snip]

> Let's consider some simple backoff strategies that gnc could
> implement, feel free to add your own.
>
> Wm's suggestion
> ===============
> only collect if the most recent price is more than 1 day (or 12 hours,
> etc) old [2].  this sort of fits in with the existing post collection
> query, i.e. always store the good prices that have been obtained,
> instead ask the user, "do you want to try and fetch the other prices /
> rates again".  this provides a natural pause to Alphavantage as gnc
> won't be hitting it for prices it just got a second time and would
> only be requesting the prices it didn't get last time.
>
> [2] I use this in my own collections (external to gnc) run by cron,
> some more obscure currency pairs are only available a few hours a day,
> I don't even try if the most recent is less than a day old.

Unless it's changed, the only real "backoff" strategy required is to
sleep for 16 seconds between requests.  Basically, assuming this hasn't
changed recently, they allow 5 requests per minute.  So by sleeping 16
seconds you ensure that 5th request occurs after 60 seconds is up.

It should be pretty easy to implement this delay and still allow full
pulls.  The downside, of course, is that if you have lots of prices to
pull it can take N/4 minutes to get them all, so sure, only refreshing
prices > 12 hours old is probably fine, but I suspect this will still be
an all-or-nothing exercise in general.

Enjoy!

-derek

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant
_______________________________________________
gnucash-devel mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Reply | Threaded
Open this post in threaded view
|

Re: [GNC-dev] Alphavantage strategies

GnuCash - Dev mailing list
On 16/01/2019 14:12, Derek Atkins wrote:

> Hi,
>
> Wm via gnucash-devel <[hidden email]> writes:
>
> [snip]
>> Let's consider some simple backoff strategies that gnc could
>> implement, feel free to add your own.
>>
>> Wm's suggestion
>> ===============
>> only collect if the most recent price is more than 1 day (or 12 hours,
>> etc) old [2].  this sort of fits in with the existing post collection
>> query, i.e. always store the good prices that have been obtained,
>> instead ask the user, "do you want to try and fetch the other prices /
>> rates again".  this provides a natural pause to Alphavantage as gnc
>> won't be hitting it for prices it just got a second time and would
>> only be requesting the prices it didn't get last time.
>>
>> [2] I use this in my own collections (external to gnc) run by cron,
>> some more obscure currency pairs are only available a few hours a day,
>> I don't even try if the most recent is less than a day old.
>
> Unless it's changed, the only real "backoff" strategy required is to
> sleep for 16 seconds between requests.  Basically, assuming this hasn't
> changed recently, they allow 5 requests per minute.  So by sleeping 16
> seconds you ensure that 5th request occurs after 60 seconds is up.
>
> It should be pretty easy to implement this delay and still allow full
> pulls.  The downside, of course, is that if you have lots of prices to
> pull it can take N/4 minutes to get them all, so sure, only refreshing
> prices > 12 hours old is probably fine, but I suspect this will still be
> an all-or-nothing exercise in general.
>
> Enjoy!

Do you really miss the point on purposes?  If gnc collects the last
price it doesn't have that is a good thing.

--
Wm
Good evening Mr Bogle.  Sleep well.


_______________________________________________
gnucash-devel mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Reply | Threaded
Open this post in threaded view
|

Re: [GNC-dev] Alphavantage strategies

Derek Atkins-3
Wm via gnucash-devel <[hidden email]> writes:

[snip]
>>> Wm's suggestion
>>> ===============
>>> only collect if the most recent price is more than 1 day (or 12 hours,
>>> etc) old [2].  this sort of fits in with the existing post collection

[snip]
> Do you really miss the point on purposes?  If gnc collects the last
> price it doesn't have that is a good thing.

All GnuCash knows is the time of the last price it received/stored.  It
has to make the request to A-V to know if there is a new price.  So that
means, by definition, GnuCash will automatically request any prices >
<Delta> old, whether or not there is a "new price", because GnuCash
can't know if there is a new price until it makes the request.

Sure, this is a "Good Thing", but it's definitely not going to limit the
number of price requests GnuCash makes....   Unless, of course, you're
assuming people are requesting prices every few minutes?

-derek

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant
_______________________________________________
gnucash-devel mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Reply | Threaded
Open this post in threaded view
|

Re: [GNC-dev] Alphavantage strategies

GnuCash - Dev mailing list
On 21/01/2019 20:10, Derek Atkins wrote:

> Wm via gnucash-devel <[hidden email]> writes:
>
> [snip]
>>>> Wm's suggestion
>>>> ===============
>>>> only collect if the most recent price is more than 1 day (or 12 hours,
>>>> etc) old [2].  this sort of fits in with the existing post collection
>
> [snip]
>> Do you really miss the point on purposes?  If gnc collects the last
>> price it doesn't have that is a good thing.
>
> All GnuCash knows is the time of the last price it received/stored.

Yup, it also can't know the significance of one price vs another for the
user.  If you're suggesting a price priority option I'm not against that
but I don't think it solves the problem you don't perceive.  It would be
a lot of work UI wise and I'm pretty sure as soon as it appeared someone
would strongly object.

> It
> has to make the request to A-V to know if there is a new price.

There is always a new price in modern markets.  That doesn't mean gnc
should be requesting a price every second or millisecond.  The db design
simply doesn't support that sort of trading.  There is other software
for that.

> So that
> means, by definition, GnuCash will automatically request any prices >
> <Delta> old, whether or not there is a "new price", because GnuCash
> can't know if there is a new price until it makes the request.

gnc can be a good citizen and not ask for a price it already has.  I am
suggesting it asks for prices it doesn't have first.  Why is this
suggestion so troubling to you?

> Sure, this is a "Good Thing", but it's definitely not going to limit the
> number of price requests GnuCash makes....   Unless, of course, you're
> assuming people are requesting prices every few minutes?

this has been done repeatedly in the user list, people *are* trying to
use gnc for intra day trading and generally find it doesn't work.

I may be misreading you, Derek
It is possible you are playing devil's advocate
It appears to me you are against gnc (or possibly F::Q) backing off and
being a good citizen or doing a good thing.

--
Wm

_______________________________________________
gnucash-devel mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Reply | Threaded
Open this post in threaded view
|

Re: [GNC-dev] Alphavantage strategies

John Ralls-2


> On Jan 22, 2019, at 1:58 AM, Wm via gnucash-devel <[hidden email]> wrote:
>
> On 21/01/2019 20:10, Derek Atkins wrote:
>> Wm via gnucash-devel <[hidden email]> writes:
>> [snip]
>>>>> Wm's suggestion
>>>>> ===============
>>>>> only collect if the most recent price is more than 1 day (or 12 hours,
>>>>> etc) old [2].  this sort of fits in with the existing post collection
>> [snip]
>>> Do you really miss the point on purposes?  If gnc collects the last
>>> price it doesn't have that is a good thing.
>> All GnuCash knows is the time of the last price it received/stored.
>
> Yup, it also can't know the significance of one price vs another for the user.  If you're suggesting a price priority option I'm not against that but I don't think it solves the problem you don't perceive.  It would be a lot of work UI wise and I'm pretty sure as soon as it appeared someone would strongly object.
>
>> It
>> has to make the request to A-V to know if there is a new price.
>
> There is always a new price in modern markets.  That doesn't mean gnc should be requesting a price every second or millisecond.  The db design simply doesn't support that sort of trading.  There is other software for that.
>
>> So that
>> means, by definition, GnuCash will automatically request any prices >
>> <Delta> old, whether or not there is a "new price", because GnuCash
>> can't know if there is a new price until it makes the request.
>
> gnc can be a good citizen and not ask for a price it already has.  I am suggesting it asks for prices it doesn't have first.  Why is this suggestion so troubling to you?
>
>> Sure, this is a "Good Thing", but it's definitely not going to limit the
>> number of price requests GnuCash makes....   Unless, of course, you're
>> assuming people are requesting prices every few minutes?
>
> this has been done repeatedly in the user list, people *are* trying to use gnc for intra day trading and generally find it doesn't work.
>
> I may be misreading you, Derek
> It is possible you are playing devil's advocate
> It appears to me you are against gnc (or possibly F::Q) backing off and being a good citizen or doing a good thing.

Wm,

F::Q has had a commit to add 6 seconds between requests in its git repo for almost a year, but the maintainer hasn’t been able to make time to do a new release. This has been discussed several times on the user list.

There’s also already a price-priority built into GnuCash. There are several levels, but the significant ones are transaction-creation, finance-quote, price-editor. Sources to the right can replace sources to their left and themselves but not sources to the right, so e.g. once  you’ve created a price in the price editor that’s the one for the day.

I don’t want GnuCash to make assumptions about what F::Q does internally, so I think the algorithm you’re proposing would look like:

1 Make list of commodities to retrieve from Alphavantage
2 Check for prices for today, splitting the above list into have-price and don’t-have-price
3 Request the don’t-have-price list. If it partly fails, wait 60 seconds and return to 2.
4 Wait 60 seconds and request the have-price list.

The code would go into libgnucash/scm/price-quotes.scm.

Regards,
John Ralls
_______________________________________________
gnucash-devel mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Reply | Threaded
Open this post in threaded view
|

Re: [GNC-dev] Alphavantage strategies

GnuCash - Dev mailing list
On 22/01/2019 15:16, John Ralls wrote:

> Wm,
>
> F::Q has had a commit to add 6 seconds between requests in its git repo for almost a year, but the maintainer hasn’t been able to make time to do a new release. This has been discussed several times on the user list.

Naughty F::Q ... don't be silly, the notion exists, it got used but not
implemented.  I don't think we (gnc and other people that use F::Q)
should presume they should protect us from being bad citizens.

And something else, the 6 seconds is growing, 16 seconds, maybe 61
seconds soon, I don't know what the back off should be but I know for
sure gnc is not suitable for trading (by that I mean prices within a day
or maybe within a few days).

> There’s also already a price-priority built into GnuCash.

 > >There are several levels, but the significant ones are
> transaction-creation,

I certainly know about that one and expect most people experience it
ordinarily in buying food, etc

> finance-quote,

I think you have the order wrong

> price-editor.

I think an actual price overrides.

>Sources to the right can replace sources to their left

my reading of what you are saying is that the price-editor should
override the actual, but I'm not convinced that is really what you mean.

>and themselves but not sources to the right, so e.g. once  you’ve created a price in the price editor >that’s the one for the day.

Hmmmn, I'm going to pause and think about that.

I think that is a wrong-thing: I think the actual human level price is
the price for the day for a person or small or modest sized organisation
rather than the price editor price, if you toss in the ordinary stuff
like some people and organisations can't get prices at the moment, what
do you think is the right price for the day?

Is it the market price (big wide world) or the price they could achieve
at the time on the day they attempted to get a price?

I am genuinely unsure about this, JohnR [99] , I think the price I have
for something must be the price for the day *because* the gnc model of
last price falls apart if it is supplanted.

[99] and other sensible commentators, remote as they may be
===

I live in London, I don't vote for Trump, if it matters I think Brexit
is dumb, I have access to market prices.

I'm generally ok with a price a person creates being prime (if I buy an
option on DogFood and Wall at USD200 per kg of federal_employee that is
my price for supporting Trump, right?) [1]

> I don’t want GnuCash to make assumptions about what F::Q does internally,

My break: I agree

I also think it will be very expensive for gnc and similar applications
that utilize F::Q to start from scratch.

> so I think the algorithm you’re proposing would look like:
>
> 1 Make list of commodities to retrieve from Alphavantage

doesn't gnc do that anyway? if that wasn't true we wouldn't be fucking
about choosing which source we wanted a quote from, FFS

> 2 Check for prices for today, splitting the above list into have-price and don’t-have-price

I think this is a good approach but it breaks if many people ask the
same question at or about the same time or repeatedly.

I don't know the answer to gnc and F::Q *not* being asked
===
alphabet share price
===
I don't use gnc or F::Q for that but I expect many people do.

> 3 Request the don’t-have-price list. If it partly fails, wait 60 seconds and return to 2.

Almost, I think the person should get a list, similar to the one
presented at presented that says,
"we couldn't get the price for
FuckwitAndCompany
MayAndIdiots
CorbynAndWeird"

but instead of asking the dumb question about storing the prices it may
have obtained

think about this

at the moment the gnc model gets good prices and throws them away!

How fucking idiotic and Trump like is that?

You have information?

What is the best thing to do?

I know!  Discard it!

Duh

Sensible people store information, prices, etc  why is gnc *discarding*
what it does get?  I simply don't understand this.  Maybe someone else
is whacking F::Q for a perfect complete set of all their data and
someone else is just trying to work out if their small business is
likely to get fucked by Brexit.

Why does gnc not store the prices it did get and then ask "do you want
me to try again, the sources may be fucked, I may not be able to get a
price again for a day or so because the fuckwit Trump is holding things
up" or whatever excuse you have but here is the question

WHY IS GNC *NOT* STORING PRICES IT DOES GET?
WHY IS GNC *DISCARDING* PRICES WHEN WE KNOW THEY ARE BECOMING MORE
EXPENSIVE?

> 4 Wait 60 seconds and request the have-price list.

if someone needs minute by minute prices I think they should pay for
that,  I'm ok with once a day at best.

> The code would go into libgnucash/scm/price-quotes.scm.

at which point it would be obscure to me.  I'd know if it worked or not
afterwards so who are you trying to impress?

===

[1] Trump has moved beyond stupid to plain funny, OK? [2]
[2] This is nothing to do with Rep or Dem any more, this is supporting
the incompetent because you're a cheap white that never had anyone to
vote for before, yay for you [3]
[3] I'm cheap white too, in case it matters [4]
[4] I'm also from africa, as in born their and tired of other whites and
their rights.

--
Wm
I am very frustrated at not being able to help the project more

_______________________________________________
gnucash-devel mailing list
[hidden email]
https://lists.gnucash.org/mailman/listinfo/gnucash-devel