OKAPI Project - API for OpenCaching sites

[oliver]

Comment from germany end user forum:
http://www.geoclub.de/viewtopic.php?p=907982

bbox in format W,S,E,N would be compatible with OpenStreeMap-XAPI

[sp2ong]

Ok we have 3 OC Nodes running with OKAPI :-)
Now http://www.opencaching.org.uk/okapi is ready

[oliver]

Code: [Auswählen]

If given cache code does not exist, the method will respond with a HTTP 400 error.

The API also responds with this code, when i dont specify cache_code in caches/geocache-call
Maybe you can choose another code?

[oliver]

Is there any way to validate the comsumer_key?
The response of the service is the same when i have invalid consumer_key or invalid cache_code.

[wrygiel]

> Another question: Does the API code require access to the real
> OC.de-source e.g. with the current upload-images-directory or is it
> possible to install the API on a second webserver? We have virtual
> server infrastructure with 2 physical hosts.

Currently, OKAPI is a plugin for OC code. It is not a standalone
application. It could be made standalone, but I am not sure if this is
a good way to go. Currently OKAPI may take advantage of the existing
OC code.

Are you quite sure making OKAPI a standalone app is a good idea?

On the other hand, relaying on existing code might be a mistake.

> services/caches/geocache
>
> - founds/notfounds ... we have a table stat_caches, so you dont
> need to recalculate these values each time. I am not sure if this
> table was introduced before or after oc-pl-merge.

I don't recalculate either, we have caches.founds and caches.notfounds
fields.

> - rating ... as i know, oc.pl has different rating system.
> recommendations sounds like the one we have.

This could be troublesome. I thought the rating system will be a
common between all OC installations. You will probably have to create
a patch for OKAPI with some "site-specific conditions".

> - images ... we have a is_hidden-flag. A user may add images that
> are only referenced in HTML for decoration. These images should not
> be shown in the image list, but may be downloaded to decorate the
> html properly.

Have you checked the OKAPI code? Here is the query I use:
http://code.google.com/p/opencaching-api/source/browse/trunk/okapi/services/caches/geocaches.php?r=80#160
Aren't you talking about the "display" column?

> services/caches/formatters
>
> - gpx ... Do you use search.gpx.inc.php for formatting or do you
> have own code?

My own:

http://code.google.com/p/opencaching-api/source/browse/trunk/okapi/services/caches/formatters/gpx.php?r=80
http://code.google.com/p/opencaching-api/source/browse/trunk/okapi/services/caches/formatters/gpxfile.tpl.php?r=80

> services/caches/search
>
> - all ... because of timeouts, memory limits or mysql locking
> issues, i think returning realy all geocaches is not a good idea. I
> dont tested it, so it may work good, but i think having no limit
> given will result to failures in future, so the API signatures
> should handle this from beginning ... could you use the LIMIT-syntax
> of mysql with start-parameter? e.g. SELECT * FROM caches ORDER BY
> wp_oc LIMIT 1000, 100

I do not return all of them. There is a limit of 500 (100 by default).
The docs were misleading.

> - size ... "caches with no container" could be size=0

Added a new issue:
http://code.google.com/p/opencaching-api/issues/detail?id=52

> - radius in /nearest ... we have a limit of 150km because
> calculation gets unefficient - then the developer should switch to
> bbox.

How inefficient? Why?

Actually, I don't like the limit. If it turned out to be inefficient,
I would try to think of some heuristics to cope with that.

> services/logs
>
> - logs should also include any limit and start-parameter. E.g. a
> smartphone-application that shows the latest logs from
> caches/geocache and has a "show more" button. Show-more will show
> the next 10 logs. If the smartphone receives all logs of the caches,
> traffic may be too much. Our cache with most finds at oc.de has 750
> finds (i've seen some at gc.com with >3000). Average log text length
> at oc.de is 300 chars. With suername, uuid and other infos, average
> bytes per log will be 400 bytes. 400 byte * 700 are 280 kByte in worst case.

You're right. But I will probably have to make this parameter
optional.
http://code.google.com/p/opencaching-api/issues/detail?id=53

> - submit ... why dont allow HTML logs?

http://code.google.com/p/opencaching-api/issues/detail?id=54

> serice/users/user
>
> - is_admin ... this should only be querable for the own user account
> for security reason.

Why? Identities of admins are hidden?

Currently I use this only in one place:
http://code.google.com/p/opencaching-api/source/browse/trunk/okapi/services/logs/submit.php?r=80#78

> Comment from germany end user forum:
> http://www.geoclub.de/viewtopic.php?p=907982
>
> bbox in format W,S,E,N would be compatible with OpenStreeMap-XAPI

I can't change it now. 2 days ago OKAPI was officially released. Now
we have to be backward-compatible.

> The API also responds with this code, when i dont specify cache_code
> in caches/geocache-call Maybe you can choose another code?

No. Have you ever used some APIs? The 400 code is a standard "bad
request" response. Both of these requests are "bad requests".

> Is there any way to validate the comsumer_key? The response of the
> service is the same when i have invalid consumer_key or invalid
> cache_code.

Which service? Does this service require authentication?

[oliver]

Currently, OKAPI is a plugin for OC code. It is not a standalone
application. It could be made standalone, but I am not sure if this is
a good way to go. Currently OKAPI may take advantage of the existing
OC code.

Are you quite sure making OKAPI a standalone app is a good idea?

No. Later, when you add function for data modification (beside "new log") e.g. change cache status or description, you will need the object from the busniess layer. With a standalone application, you will have duplicate validation logic.

What i mean is: Create a clean copy of the OC-source on a second host (api-host) and install OKAPI there. On that copy, you will have no file system access to user uploaded images, user stat-pictures etc. But the user app can load that images with a http-request to the www-host. So i dont think thats a real problem.

[wrygiel]

> No. Later, when you add function for data modification (beside "new
> log") e.g. change cache status or description, you will need the
> object from the busniess layer. With a standalone application, you
> will have duplicate validation logic.

I think so too.

> What i mean is: Create a clean copy of the OC-source on a second
> host (api-host) and install OKAPI there. On that copy, you will have
> no file system access to user uploaded images, user stat-pictures
> etc. But the user app can load that images with a http-request to
> the www-host. So i dont think thats a real problem.

OKAPI does not store any cached data in the file system. But, on there
other hand, your "business layer" - might.

[oliver]

This could be troublesome. I thought the rating system will be a
common between all OC installations. You will probably have to create
a patch for OKAPI with some "site-specific conditions".

I would prefer a "feature configuration file" in official repository code.

$opt['have_rating'] = true|false;
$opt['have_cache_found_stat'] = true|false;

> - images ... we have a is_hidden-flag. A user may add images that
> are only referenced in HTML for decoration. These images should not
> be shown in the image list, but may be downloaded to decorate the
> html properly.

Have you checked the OKAPI code? Here is the query I use:
http://code.google.com/p/opencaching-api/source/browse/trunk/okapi/services/caches/geocaches.php?r=80#160
Aren't you talking about the "display" column?

No, i did not check the source code until now.

> services/caches/formatters
>
> - gpx ... Do you use search.gpx.inc.php for formatting or do you
> have own code?

My own:

http://code.google.com/p/opencaching-api/source/browse/trunk/okapi/services/caches/formatters/gpx.php?r=80
http://code.google.com/p/opencaching-api/source/browse/trunk/okapi/services/caches/formatters/gpxfile.tpl.php?r=80

okay.

> services/caches/search
>
> - all ... because of timeouts, memory limits or mysql locking
> issues, i think returning realy all geocaches is not a good idea. I
> dont tested it, so it may work good, but i think having no limit
> given will result to failures in future, so the API signatures
> should handle this from beginning ... could you use the LIMIT-syntax
> of mysql with start-parameter? e.g. SELECT * FROM caches ORDER BY
> wp_oc LIMIT 1000, 100

I do not return all of them. There is a limit of 500 (100 by default).
The docs were misleading.

What do you think about the "start-at"-paramter?

(i will continue when i am back at home)

[oliver]

> What i mean is: Create a clean copy of the OC-source on a second
> host (api-host) and install OKAPI there. On that copy, you will have
> no file system access to user uploaded images, user stat-pictures
> etc. But the user app can load that images with a http-request to
> the www-host. So i dont think thats a real problem.

OKAPI does not store any cached data in the file system. But, on there
other hand, your "business layer" - might.

Ah, once you will add "image upload" feature (maybe iphone-app "log with picture), we will need a solution for that. But your answer is good, then i can install it on a new host with PHP 5.3 support when we fixed the compatibility issues.

[wrygiel]

> I would prefer a "feature configuration file" in official repository code.
>
> $opt['have_rating'] = true|false;
> $opt['have_cache_found_stat'] = true|false;

Okay. I will implement this on Monday or Tuesday.

> What do you think about the "start-at"-paramter?

It depends on context.

I wouldn't encourage developers to download our entire database. OKAPI
is meant primarilly as a tool for online access. If developers want to
have dumps of data for offline use, I think we should think of a
special method for that - it would return a static JSON/XML/SQL dump,
generated once a day.

[oliver]

Okay, back at home

I tried to implement VB.NET client library yesterday night and stucked when coming to oauth-login ...

> - radius in /nearest ... we have a limit of 150km because
> calculation gets unefficient - then the developer should switch to
> bbox.

How inefficient? Why?

I am not sure. I tested some things in 2005 and result was "limit it to 150km and you have no problem". For larger search radius, a bbox-search is most often appropriate, too. You know, for real radius search, there is a lot of sin() and cos()-calculation and processing time grows in square.

Most efficient way for distance calculation i found for mysql:

SELECT %distance AS distance {...} FROM caches WHERE lat>%min_lat AND lat<%max_lat AND lon>%max_lon AND lon<%max_lon HAVING distance<%max_distance ORDER BY distance ASC

where:
%distance is the formula returned by getSqlDistanceFormula()
%min_lat is the value returned by getMinLat()
%max_lat is the value returned by getMaxLat()
%min_lon is the value returned by getMinLon()
%max_lon is the value returned by getMaxLon()

With that formular, mysql performs a bbox-search first. That is fast. And only for that results, it calculates the distance formula. That is accurate.

Would be nice to that query pattern. In germany we already had some database performance issues in burst times before we offloaded search queries to our mysql slave server (on the second physical host).

> serice/users/user
>
> - is_admin ... this should only be querable for the own user account
> for security reason.

Why? Identities of admins are hidden?

In general, we do not hide the real accounts behind our admins as groundspeak does. But i think its not required to make the is_admin queryable for everyone and in an bruteforce-attack, the first goal is to find a priviledged account. I have no problem if the API reports that the logged in user has admin priviledges.

Currently I use this only in one place:
http://code.google.com/p/opencaching-api/source/browse/trunk/okapi/services/logs/submit.php?r=80#78

Ah, okay ... in the new code, there is a cache-business-layer-class which has property allowLog(). It would be a dream to have one code for all nodes

> Comment from germany end user forum:
> http://www.geoclub.de/viewtopic.php?p=907982
>
> bbox in format W,S,E,N would be compatible with OpenStreeMap-XAPI

I can't change it now. 2 days ago OKAPI was officially released. Now
we have to be backward-compatible.

If you think the suggestion of the geoclub-user is good, you could add a second supported input format:
- existing one with | seperated
- new one with , seperated

This should be backward-compatible.

> The API also responds with this code, when i dont specify cache_code
> in caches/geocache-call Maybe you can choose another code?

No. Have you ever used some APIs? The 400 code is a standard "bad
request" response. Both of these requests are "bad requests".

I have used lots of APIs. And every good API gives a precise reason why the call failed. However, i did not use any JSON/REST-API until now. I worked with SOAP exceptions-model.

> Is there any way to validate the comsumer_key? The response of the
> service is the same when i have invalid consumer_key or invalid
> cache_code.

Which service? Does this service require authentication?

It was services/caches/geocache

Thank you for accepting most of my points as an "issue". I know it is many "should and could and i would do in another way" - but again, you API looks good and i will try to get it running with our new code. But i need to be sure that i dont install any code at oc.de that does not fully comply with our busines logic.

I want to test some calls with my .NET-client-library (and test oauth) and then i will try to assist you with the "feature configuration file". solar22 is in vacation for a few weeks, as i remember.

[wrygiel]

> I tried to implement VB.NET client library yesterday night and
> stucked when coming to oauth-login ...

Yeah, OAuth isn't easy.

> In general, we do not hide the real accounts behind our admins as
> groundspeak does. But i think its not required to make the is_admin
> queryable for everyone and in an bruteforce-attack, the first goal
> is to find a priviledged account. I have no problem if the API
> reports that the logged in user has admin priviledges.

Okay. You convinced me.
http://code.google.com/p/opencaching-api/issues/detail?id=57

>>> bbox in format W,S,E,N would be compatible with OpenStreeMap-XAPI
>
>> I can't change it now. 2 days ago OKAPI was officially released. Now
>> we have to be backward-compatible.
>
> If you think the suggestion of the geoclub-user is good, you could
> add a second supported input format:
> - existing one with | seperated
> - new one with , seperated
>
> This should be backward-compatible.

True, but it would be REALLY ugly.

>>> The API also responds with this code, when i dont specify
>>> cache_code in caches/geocache-call Maybe you can choose another
>>> code?
>
>> No. Have you ever used some APIs? The 400 code is a standard "bad
>> request" response. Both of these requests are "bad requests".[/quote]
>
> I have used lots of APIs. And every good API gives a precise reason
> why the call failed. However, i did not use any JSON/REST-API until
> now. I worked with SOAP exceptions-model.

But OKAPI *does* respond with a detailed error message! 400 is only a
HTTP response code, the message is the response *content*.

>>> Is there any way to validate the comsumer_key? The response of the
>>> service is the same when i have invalid consumer_key or invalid
>>> cache_code.
>
>> Which service? Does this service require authentication?
>
> It was services/caches/geocache

I don't believe you What was this response exactly? These are
two completelly different errors and I'm quite sure they are reported
with similarly different error messages.

BTW, I am also not home. I will be more available after Monday.

[oliver]

> I have used lots of APIs. And every good API gives a precise reason
> why the call failed. However, i did not use any JSON/REST-API until
> now. I worked with SOAP exceptions-model.

But OKAPI *does* respond with a detailed error message! 400 is only a
HTTP response code, the message is the response *content*.

Ah, okay! Then i have to look, how to get the resoponse-content in .NET Framework, after an Exception was thrown. I checked all properties of the .NET exception object, but did not find it. Thank you for pointing me to that.

[sp2ong]

I have see that application c:geo is opensource https://github.com/cgeo/c-geo-opensource and maybe it is easy add ? OKAPI support to this application

[oliver]

Ah, okay! Then i have to look, how to get the resoponse-content in .NET Framework, after an Exception was thrown. I checked all properties of the .NET exception object, but did not find it. Thank you for pointing me to that.

Works now
Thank you!