Opencaching Community

Cachen auf opencaching.de => Geocaching-Tools => Thema gestartet von: tomaszg am 28. Januar 2020, 03:28:20

Titel: Stats for Opencaching sites
Beitrag von: tomaszg am 28. Januar 2020, 03:28:20
Hi,

I'm developing a tool to compute various statistics related to Opencaching activity and a cache map. It supports all OC sites which use OKAPI. The tool itself is accompanied by a simple webpage to visualize the results and you can check it out here: http://tomaszg.pl/geo/.

It is still work in progress and especially all international features were not extensively tested. Myself, I use only OC.pl so many things may break. Also I still intend to add new features, so all ideas are welcome.

If anyone is interested to contribute, the source code is hosted at https://gitlab.com/tomaszg/geostat.
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 28. Januar 2020, 06:25:29
Hi Tomasz, it seems that this year 2020 ist the year of statictics.
We have already two statictics for oc.de available ;)

Regarding Your website: My nick (mic@) is not accepted:
==> Incorrect characters in nick d.
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 28. Januar 2020, 06:44:19
Regarding Your website: My nick (mic@) is not accepted:
==> Incorrect characters in nick d.

Ah, didn't expect a "@". Should work now.

BTW. Your German language question captcha is a bit annoying ;)
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 28. Januar 2020, 06:54:27
Zitat von: tomaszg
Ah, didn't expect a "@". Should work now.

Wow, thx for the fast reaction.
The statistic is very beautiful !!!

Is there a chance to get an overall statistic, for example a heatmap of ALL oc.de users?
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 28. Januar 2020, 10:29:26
Is there a chance to get an overall statistic, for example a heatmap of ALL oc.de users?
Do you mean just a heatmap of cache locations or heatmap of finds? In any case OKAPI doesn't allow it easily. It limits the amount of data user can pull. I would have to clone the database on my end and work with it. I'm doing that for OC.pl to plot caches not found but it is slow and requires storing the data locally on my server. I could try to do it. For OC.pl it is easy to get the cache heatmap by checking "not found" toggle and specifying a user without any finds (or with minimal one).

BTW. I just realized that OC.de hosts a lot of "Safari" reverse caches. I should probably try to exclude them from geographic part of stats. Too bad they don't have a separate cache type... I explicitly excluded "own" and "mobile" type of caches already.
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 28. Januar 2020, 14:02:13
I downloaded the data dump and rendered a cache density heat map:

(http://tomaszg.pl/geo/geo_de.jpg)

(http://tomaszg.pl/geo/geo_de_eu.jpg)
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 28. Januar 2020, 21:31:38
Zitat von: tomaszg
Do you mean just a heatmap of cache locations or heatmap of finds?

Correct.
So the result would be something like this here:
http://www.flopp.net/heatmap
Unfortunately this heatmap is only for caches, not for finds.


Zitat von: tomaszg
BTW. I just realized that OC.de hosts a lot of "Safari" reverse caches. I should probably try to exclude them from geographic part of stats. Too bad they don't have a separate cache type... I explicitly excluded "own" and "mobile" type of caches already.

All safari caches have a specific attribute.
Maybe this could help for building the right filter.
And "maybe" we will fix this problem soon, because there is the idea of
simplifying the cache types (e.g. remove math caches and introduce safari cache).

Your maps look great - thank You very much!
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 29. Januar 2020, 18:41:33
Correct.
So the result would be something like this here:
http://www.flopp.net/heatmap
Unfortunately this heatmap is only for caches, not for finds.
If I find the time, I'll try to prepare something for finds. I'll probably only upload a generated image, but it might be fun to create a web interface to be able to trace geocaching activity in a given time period.

All safari caches have a specific attribute.
Maybe this could help for building the right filter.
And "maybe" we will fix this problem soon, because there is the idea of
simplifying the cache types (e.g. remove math caches and introduce safari cache).
I noticed that. My code is not yet ready to handle attributes but when it does I'll try to filter them out.
Titel: Re: Stats for Opencaching sites
Beitrag von: FriedrichFröbel am 01. Februar 2020, 10:46:49
By the way: It seems like umlauts are not allowed inside the user name as well at the moment.
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 01. Februar 2020, 11:59:52
Now it should work. I managed to get better unicode support in my web script and I hope I didn't break anything else by doing that.
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 29. Februar 2020, 14:24:40
Added a bit of stuff last month: display of "trail" instead of heatmap (works so-so), time range (i.e. limiting stats to caches found in a certain period), added stars next to names of recommended caches, and a few minor polishing things.
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 29. Februar 2020, 17:02:13
Thank You for your great wirk.

I just trid to view my statictics again, but it failed:
OKAPI error: "Parameter 'cache_codes' has invalid value: Duplicate codes detected (make sure each cache is referenced only once)."

I tried it with "mic@" and german map
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 29. Februar 2020, 23:08:08
It seems I reintroduced an old bug with these changes. Should be fixed now.
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 01. März 2020, 00:10:47
Zitat von: tomaszg
Should be fixed now.

Yes, it works smoothly again. Thank You for Your work!
Normally I am not a big statistic fan, but it makes fun to watch my caching behaviour in numbers and caches.
Very cool!
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 01. März 2020, 22:43:27
Is it possible to allow statistics only for the owner himself/herself?

Actual I am possible to view any oc-user just by replacing USERNAME with the correct user (e.g.: zelda):
http://tomaszg.pl/cgi-bin/geo-form.pl?nick_de=USERNAME&stamp=15&map=Germany
turns to
http://tomaszg.pl/cgi-bin/geo-form.pl?nick_de=zelda&stamp=15&map=Germany

So it´s a bit like "big brother is watching you"...
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 02. März 2020, 00:02:37
Well, I can't do anything about it, it's Opencaching which is open and shares all the data with everybody :) I'm not asking anyone their OC password to identify them. It might be possible to devise some way to do it (not easily for me), but that wouldn't change the fact that all this data is for anyone to grab. Just entering somebody's OC profile you can see where and when he was in the past just by listing caches found. It even offers the possibility to export this data in easily parsable form, like gpx or kml. Simply dropping it into Google Maps would yield similar result.

Bottom line - it is something which could be resolved in a meaningful way only on OC end
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 24. Mai 2020, 18:50:52
Zitat von: tomaszg
I'm developing a tool to compute various statistics related to Opencaching activity and a cache map. It supports all OC sites which use OKAPI. The tool itself is accompanied by a simple webpage to visualize the results and you can check it out here: http://tomaszg.pl/geo/.

Is it possible to offer this tool on https too?
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 24. Mai 2020, 19:04:46
Sorry, my server doesn't support it. I never learned how to set it up properly. I'll look into it if I find the time :)
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 24. Mai 2020, 19:54:45
Zitat von: tomaszg
Sorry, my server doesn't support it.

Have You ever talked to oc.pl team? They use https  8)
https://www.opencaching.pl
Maybe they could inherit Your tool in their code?
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 24. Mai 2020, 20:18:27
In principle, everybody is free to deploy the tool wherever they wish. It is enough to grab the source and compile it.

I've been in contact with OCpl team. They are always looking for people to work on the service. It would be best just to incorporate it with the stats on the webpage. Regretfully that would require knowledge of PHP and I don't know enough to do it. My tool is written in C++.

I didn't consider the lack of https before to be relevant to the webpages I have. I just started the investigation and it might turn out to be quite simple. I'll need to contact IT support before though and if they reply quickly enough, I may have it working tomorrow.
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 29. Mai 2020, 23:17:58
Okay, it should work now with https.
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 29. Mai 2020, 23:43:36
Okay, it should work now with https.

Perfect  8)

One more thing:
When I go to https://tomaszg.pl/geo and want to start with myself (nick=mic@ / map=Germany)
then the result is:
https://tomaszg.pl/cgi-bin/geo-form.pl?nick_pl=&nick_de=mic%40&nick_us=&nick_nl=&nick_ro=&nick_uk=&stamp=15&start_date=&end_date=&map=Germany

But
https://tomaszg.pl/cgi-bin/geo-form.pl?nick_de=mic%40&stamp=15&map=Germany
works the same...

So my question is: Could You erade useless information like &nick_nl=&
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 30. Mai 2020, 00:02:18
I would need to plug in some javascript to do it, as this is normal behaviour for HTML. I'm reluctant to use JS for such trivial thing, even more so since it is purely aesthetic :)
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 30. Mai 2020, 08:31:10
Zitat von: tomaszg
I would need to plug in some javascript to do it

I agree, no need for JS. My wish was only a "nice-to-have" feature for a smaller URL, but it works with long URL too.
THX again for all Your work, its brilliant.
Maybe You should call it Project OC  8)

Best wishes, Mic@
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 03. Juni 2020, 22:56:00
One minor bug. Please take a look at the Difficulty / terrain matrix:
https://tomaszg.pl/cgi-bin/geo-form.pl?nick_de=zelda&stamp=15&map=Germany
The calculated values (Total 26 combinations found out of 81) are wrong.
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 04. Juni 2020, 00:11:20
Thanks for spotting! It was caused by simple copy/paste error, fixed now.
Titel: Re: Stats for Opencaching sites
Beitrag von: mic@ am 24. Juni 2020, 12:08:11
Zitat von: tomaszg
Okay, it should work now with https.

Have You stepped back to http?

Because http://tomaszg.pl/cgi-bin/geo-form.pl?nick_de=mic%40&stamp=15&map=Germany  works,
but https://tomaszg.pl/cgi-bin/geo-form.pl?nick_de=mic%40&stamp=15&map=Germany is not working anymore?!
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 24. Juni 2020, 12:11:41
I have some temporary configuration problems and I'm now working to fix it. Broke something last night, it seems.

Edit: I think everything is fixed now.
Titel: Re: Stats for Opencaching sites
Beitrag von: JoDaEmPA am 02. Juli 2020, 23:44:46
HeyHo !
Thangs tomaszg ! It a very cool project ! Stats for Opencaching !
Can and do you want to install a filter for all FTF hunters that counts the FTF ?

https://project-gc.com/ProfileStats/jodaempa#FTF

Great Thangs JoDaEmPa !
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 04. Juli 2020, 21:53:10
I wanted to implement some FTF stats, but regretfully it is rather difficult. I can't in a straightforward way check it via Okapi without parsing all log entries. I'm trying to find a way to do it in a reasonable way, but didn't figure it out yet.

FWIW, Project-GC checks for word "FTF" in the log to figure it out. I don't think many people put it in their logs (I know I don't usually) so it is not the best approach.
Titel: Re: Stats for Opencaching sites
Beitrag von: j-he am 04. Juli 2020, 22:56:03
FWIW, Project-GC checks for word "FTF" in the log to figure it out. I don't think many people put it in their logs (I know I don't usually) so it is not the best approach.
As an alternative, Project-GC lets you use and configure a public Bookmark List for your FTFs, which is the method I use. I think it is more reliable than looking at log entries, as there may be other reasons to write about FTF in a log entry.
Titel: Re: Stats for Opencaching sites
Beitrag von: tomaszg am 05. Juli 2020, 00:34:20
Right, but I'm not sure what would be an alternative of this approach for OC. I see you have some kind of lists on OCDE which could be used for this end. On the other hand, we don't have such a thing on OCPL as far as I know.  I would prefer to have some solution that would work across the board... Uploading a file with a custom list of cache codes would be tedious.
Titel: Re: Stats for Opencaching sites
Beitrag von: JoDaEmPA am 13. Juli 2020, 12:05:34
At
project-gc.com also uses a filter with {FTF} entries!
But the method of j-he to put the {FTF's} in a bookmark list is probably the easiest method!
Thanks tomaszg
{translated with Google}