Creating Web Services
Note: NetTalk Web Services requires xFiles
Introduction
There are a variety of common features that programs can offer that
collectively are known as Web Services. NetTalk 8 introduced two new
Procedure templates, called NetWebService and
NetWebServiceMethod to assist
with creating web services.
A NetWebService
is the name for a collection of service methods.
In that
application tree you can have multiple NetWebServiceMethods for a single
NetWebService. The NetWebService is just a container, and does not
include much more than just a list of the methods.
Your
application can contain multiple NetWebService procedures (each one with
one or more methods.)
All methods support a variety of encodings,
for input and for output. Possible input encodings are XML (using xFiles),
JSON (using jFiles) or Form-URL-Encoded. Possible output encodings are
XML or JSON.
Supported Web Service standards include WSDL, SOAP
and REST.
Example
In NetTalk 7 and earlier a hand-coded approach to web services was
demonstrated in an example called SOAPServer (42).
This example is included for reference purposes.
However a new
example, WebService (77) is provided from NetTalk 8
and as it makes use of the new templates, is a better example for
developing future WebServices.
Documentation
The service, and each method is self documenting. From any browser you
can see the documentation for the service and for each method. The
documentation includes as much information as possible so that other
developers can easily make use of your service.
To see the
documentation in your browser navigate to one of the following links;
www.whatever.com/servicename
www.whatever.com/servicename?help
www.whatever.com/methodname?help
www.whatever.com/servicename?methodname
WSDL
A WSDL file is a formal, computer-readable way of documenting a web
service. Using this file, other programmers can write programs that make
use of your service, and the existence of this file makes their lives
much easier.
NetTalk automatically (and dynamically) generates
this file for you when it is requested. A WSDL file for your service can
be retrieved by using one of these links;
www.whatever.com/servicename?wsdl or
www.whatever.com/methodname?wsdl
Method Data
The method template has a place where you can set the incoming data fields, and the returning
data fields. These fields need to be declared in your application either
as tables, table fields, global data or local data.
Incoming fields can
be any simple data type or a TABLE, GROUP or QUEUE structure. (If a QUEUE
structure is used then
declaring it as a local queue is recommended - global queues are strongly
discouraged.)
Like a normal Clarion procedure the method can take
multiple incoming parameters and (depending on your code) the caller may
only need to send some of them - in other words some of the incoming
parameters may be optional.
If a Table is selected as an incoming
parameter, then another parameter (
tablename_action) is also
expected from the caller. The action parameter informs the server of the
nature of the change or fetch.
The template includes validation
options for the parameters, but you can also add your own hand-code
validation where necessary.
Unlike a normal Clarion procedure,
the method can return multiple variables. Return values can be simple
fields (including Local or Global variables), Queues, Groups, Tables and
Views.
All methods can return one or more errors in place of the
declared return values. If any errors exist, then none of the
other declared variables are returned. For more on errors see
Errors.
All methods can also make use of
the standard
ServiceResult queue. The
template code will especially make use of this when an incoming Table
parameter is used. For more on standard results see
Results.
Fields in Returning VIEWs
On of the Return types supported is a VIEW. Views are particularly
useful here because they allow you to send a subset of a table (ie
rows, or columns in the table can be suppressed.)
By default
all
[1] the fields in the Table are included in
the view. You can override this by adding only the fields you want
to export to the View Fields list in the template. The fields will
be formatted according to the dictionary settings, unless this
feature is disabled for this method, or globally.
[1] All is perhaps not quite true. You can set
fields in the dictionary so they will not be included.
If the
Field User Option
NETAPI exists, and is
set to
0, then this field will not be
included. If it is set to
1 then the
field is included (even if it is over another field.)
Using
the
NETAPI switch allows you to
expressly exclude the "parent" field of the OVER, and expressly
include the "child" field of the OVER. It can be very important to
do this if you have a GROUP over a STRING.
Consider the
following case;
ArriveStamp STRING(8)
ArriveStampGroup GROUP,OVER(ArriveStamp)
ArriveDate DATE
ArriveTime TIME
END
In this situation, by default, the ArriveStamp field
would be automatically included. This is almost certainly not
desirable though because "Binary" data is not allowed in XML (it's
ok in Json) and because unless the client is a Clarion program this
string is meaningless anyway.
In this case it would be
advantageous to set
NETAPI to 0 for
ArriveStamp.
Tables
Probably the most common use
for a WebServiceMethod is to read and/or write database records. Since
this is a common use-case, the templates are aware of this situation,
and creating these sorts of methods is very straightforward.
- Create a NetWebServiceMethod procedure. Add it to a
NetWebService as normal.
- Create an incoming parameter of type TABLE, and select a Table.
Tick on the database actions you want to allow (Inserts, Updates etc)
- If the method allows the user to read the table then create a
return value, of type VIEW. Give the view a name, tick on
GenerateViewStructure and select the Table to view. If you leave the
Fields list blank then all the fields from the table will be
exported. Alternatively you can select a subset of fields to send to
the client. Also be sure to enter an appropriate filter here.
The service will look for an additional parameter, called
TableName_Action.
This parameter is a string and should contain one of
insert,
update,
get or
delete.
Wizard
The wizard can now generate a web service and
a web service method for each table that you select.
Calling Methods from a Client program
The above calls are typically made from a browser, and the goal is to
determine what the service can do, and how to use it. To actually make use
of the service though you need to call one of the methods.
AA service is a collection of one or more methods. You can create your
own methods using the
NetWebServiceMethod procedure type.
You can add as many methods as you like to a service, and your application
can contain as many services (ie collections of methods) as you like.
Methods
have a name, a list of incoming parameters, and a list of returned data.
While a Clarion procedure only returns a single piece of data, a web method
can return any amount of data, including GROUP and QUEUE structures.
The
NetWebServiceMethod
procedure type supports a number of calling techniques;
-
Normal HTTP GET. The URL is of the form
/service/method. Incoming parameters are passed as part of the URL or
as cookies. The returned result is a simple XML structure. For example;
GET /School/GetSchoolTeacher?FromTeacherID=value&ToTeacherId=value&Authenticate=value
It's also possible to call the method using the PUT
or DELETE commands instead of GET. For
example;
PUT /School/GetSchoolTeacher?TeacherID=value&Teacher
Name=value&
In the method itself you can determine which Verb was
used by checking p_web.RequestMethodType.
Typical possible values are one of NetWebServer_GET,
NetWebServer_PUT,
NetWebServer_POST and NetWebServer_DELETE.
- Normal HTTP POST. The URL is of the form
/service/method. Incoming parameters are passed as Post Data, but in
addition to the URL and Cookies, data can be passed in as plain "POST Data".
For example;
POST /School/GetSchoolTeacher HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: length
FromTeacherID=value&ToTeacherId=value&Authenticate=value
- SOAP 1.1. The URL is of the form /service.
In this case a POST is used, but the incoming Post Data is formatted as XML,
and (optionally) wrapped in a SOAP envelope. A HTTP Header called
SOAPAction: is also set to the method name.
POST /School HTTP/1.1
Host:
somehost
Content-Type: text/xml
Content-Length: length
SOAPAction: /GetSchoolTeacher
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<GetSchoolTeacher xmlns="http://www.capesoft.com">
<FromTeacherID>value</FromTeacherID>
<ToTeacherId>value</ToTeacherId>
<Authenticate>
<User>value</User>
<Password>value</Password>
</Authenticate>
<AsAtDate>value</AsAtDate>
<AsAtTime>value</AsAtTime>
</GetSchoolTeacher>
</soap:Body>
</soap:Envelope>
- SOAP 1.2. The URL is of the
form /service. the request is very similar to a
SOAP 1.1 request, although the SOAPAction header is not included. the
content-type of the SOAP 1.1 request is text/xml whereas the content-type
for a SOAP 1.2 request is application/soap+xml. For example;
POST
/School HTTP/1.1
Host: somehost
Content-Type: application/soap+xml
Content-Length:
length
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
<soap:Body>
<GetSchoolTeacher xmlns="http://www.capesoft.com">
<FromTeacherID>value</FromTeacherID>
<ToTeacherId>value</ToTeacherId>
<Authenticate>
<User>value</User>
<Password>value</Password>
</Authenticate>
<AsAtDate>value</AsAtDate>
<AsAtTime>value</AsAtTime>
</GetSchoolTeacher>
</soap:Body>
</soap:Envelope>
Your Code in the method
Services can include automatically generated methods (more on that in a
moment) but a NetWebServiceMethod procedure doesn't actually do anything
unless you add the necessary code.
In this sense a method can do
"anything" that you code it to do. The template will parse the incoming
request, and place it in the Parameter data structures. It will also
format the return structures into XML or JSON, and return them to the caller.
Your job is to write the code that populates the return structures.
You add your code to a routine called ServiceMethod. By the time
this routine is called the parameter structures have been primed. Once
this code is completed another (generated for you) routine will turn the
result data structures into XML (or JSON) - you don't need to worry about that,
you just write the code to populate the return structures with the
correct value. Your code should not need to care about the technique
used to call the method.
You add normal Clarion code here,
opening tables, reading data and performing calculations, just like you
would in any Clarion procedure.
You have access to the
SessionQueue here for fetching and storing information using the normal
p_web.GetSessionValue and
p_web.SetSessionValue methods. However the
incoming request will be bound to the session ONLY if the
sessionID cookie is set in the request. Since the web client accessing this method
is usually not a browser, the cookie may not automatically be set.
Aside: If you are using a Clarion program as a client, and you are
using the NetWebClient class, then you can set the
NetWebClient.OptionAutoCookie property so that multiple
requests will preserve the cookies, and hence the session ID. If you are
using a different tool for the client, then you will need to research
that tool to determine how to send the cookie.
If your code does
not interact with the session queue, then you don't need to worry about
this.
Your code can (and should) generate Errors when things go
wrong. See the next section for more on adding errors to your code.
Results
NetTalk contains a generic Results queue (
p_web.ServiceResultQueue)
which is used to pass information back to the caller. This queue is
populated when the caller is adding, editing or removing records in the
database, using the generated template code.
The Queue contains
four fields;
- Action
- TableName
- RecordID
- Description
You can add your own information to the queue
if you wish. The method to call is
p_web.AddServiceResult (action, tablename, recordId, description)
Errors
A method may fail for any number of reasons. Services should return detailed, meaningful
error information to the client wherever possible. To this end NetTalk Service Methods include
a built-in, automatic, always consistent queue of errors.
Errors
can be generated automatically by the validation template settings, or
they can be generated in your method code. To Generate a method simply
call;
p_web.AddServiceError
(Number, Position, RecordID, Description, Recommendation)
You are free
to pass whatever you like to the AddServiceError
method, but the more information passed to the client the better.
If any errors are added to the queue in this way, then only the
errors list will be returned to the caller. None of the other return
values will be included in the reply.
The ServiceErrorQueue
is automatically added as a possible reply to the generated WSDL file
for all methods in the service.
SOAP versus REST
As you can see from the above your method will happily accept incoming
requests formatted as a SOAP XML packet. It is equally happy though to
receive the request as a simple GET, PUT, POST or DELETE command. This
is sometimes known as a REST request.
Your embed code does not
change greatly between creating a SOAP service, or a REST service. REST
is basically the same as SOAP without all the SOAP wrapping. Typically a
REST client will also use the different HTTP verbs (GET, POST, PUT and
DELETE) to match up to regular file activities. A SOAP service on the
other hand will typically use a parameter to determine the file action
to take.
The template generated code can easily handle both, for example in your
embed code you might have some code like this;
(In this example a
parameter called ACTION is assumed. If the
parameter is sent, then loc:act is set from
that, if the parameter is omitted, or set to 0, then the HTTP verb is
checked and the loc:act based on that.)
loc:act = action
if loc:act = 0
case p_web.RequestMethodType
of 'GET'
loc:act = Net:ViewRecord
of 'DELETE'
loc:act = Net:DeleteRecord
of 'PUT'
loc:act = Net:ChangeRecord
of 'POST'
loc:act = Net:InsertRecord
end
end
To be a truly RESTful method you should not need to access the SessionQueue in
order for the method to work. Ideally the client should pass you all the
information you need in order for the method to work.
Return Format Rules
NetTalk API's can return results in either XML or JSON format. NetTalk applies a number of rules to
decide which format to use. As soon as a rule is met then the decision
is made, and no further rules are tested. The rules are as follows;
- If JSON is not supported by the method then return XML.
- If XML is not supported by the method then return JSON.
- If the incoming ACCEPT header specifies XML then return XML.
- If the incoming ACCEPT header specifies JSON then return JSON.
- If the incoming request type is SOAP 1.1 or SOAP 1.2 then return
XML.
- If the incoming CONTENT-TYPE header is XML then return XML.
- If the incoming CONTENT-TYPE header is JSON then return JSON.
- If the
local template setting is not DEFAULT then use local
template setting
- Use
global
template setting
Overriding XML or JSON methods
The NetWebServiceMethod will generate two objects[1] in the procedure.
The one (xml) supports XML structures, the other (json) supports JSON
structures.
xml xFileXML
json
JSONClassThese are simple object declarations based on
the xFiles
xFileXML and the jFiles
JSONClass classes respectively. These
objects are used internally to parse incoming requests, and to create
outgoing answers.
There may be times when it is desirable to
override one or more methods in one, or both, of these objects. For
example when parsing xml input it may be necessary to add embed code to
the
xFileXML.AssignField method (as
described in the xFiles
documentation.) To expose the embed points it is necessary for the
object to rather be generated by the appropriate extension template.
For XML, go to the Extensions tab and add the xFiles
IncludexFilesObject extension. Set the
object name to
xml. (This is important, it MUST have
this name.) Then go to the settings for the method (Properties /
Actions), to the General tab, and turn off the
Generate XML
Object option there. Doing these two steps means the xml object
declaration is generated by the xFiles template, and not generated by
the NetTalk template. Once this is done all the embed points for the xml
object will be defined and you can embed code in them.
For JSON
you follow the same process as for XML, but add the jFiles
IncludejFilesObject extension and set the
object name to
json. Then on the General tab turn off
the
Generate JSON Object setting.
[1] The JSON
object is only generated if the jFiles global extension has been added
to the app.
Formatting Fields in the Response
From build 9.17, support for automatic formatting
and
deformatting of
parameter and return fields is included.
This feature can be activated
and deactivated
globally. (It is on by default.)
It can be overridden at the
field level in the Method.
Specifically fields which are
stored as "not String" and have a picture, are automatically converted to strings (using
the picture) and vice-versa.
For Parameters this feature is limited
to parameter types TABLE, DATE, TIME, NUMBER and STRING.
GROUP, QUEUE, STRINGTHEORY and FILE are not supported.
For Return values this feature is
limited to FIELD (from the dictionary, not local data), TABLE and
VIEW return types. For VIEWs only Generated View fields are done
automatically.
Embed points exist for both XML and JSON
which allow you to format, and deformat fields in hand-code.
Format | Type | Method | Code Example |
xml | parameter | xml.AssignField |
Self.CurrentField = Deformat(pString,'@d6') |
json | parameter | json.DeformatValue |
Return Deformat(pValue,'@d6') |
xml | return value | xml.SaveCurrentFieldToXML |
self.FormatCurrentField('@d6') |
json | return value | json.FormatValue |
pLiteralType
= json:string Return clip(left(format(pValue,'@d6')))
|
For disconnected sync the server, and the desktop need to
be speaking the same language. For this reason the Desktop Sync
client now defaults to auto formatting. If you disable auto
formatting on the server side, then it needs to be disabled on
the Desktop Client as well.
Currently the JavaScript sync
is always formatted, so it is recommended that sync methods
leave auto formatting on.
For more information on formatting the output of a
field in xFiles, see here
http://www.capesoft.com/docs/xFiles/xfiles.htm#FormattingSaveField
For more information on formatting the output of a field in jFiles
see here
http://www.capesoft.com/docs/jfiles/jfiles.htm#FormattingSavedValues
Field Prefixes
Field prefixes are a thorny problem for a couple of reasons.
Firstly, they're a foreign concept in most languages, so they are
often not desirable in a public API at all. Secondly they make use
of a colon separator, which is not a valid character in a variable
name in most languages.
For this reason you have the option
to include, or exclude prefixes from parameters and return values.
In most cases having the prefix OFF seems the better option.
For builds prior to 9.17 the default for prefixes was ON. For
parameters and values added using 9.17 or later the default value is
OFF. the setting for existing parameters and returns are not changed
by the update, however support for the prefix support in the system
has been overhauled so definitely check that your
methods are still working the way you expect.
When
writing a web app, colons are translated into a double underscore to
make the names compatible with HTML. For
ServiceMethods though, colons are translated into a single
underscore. In builds 9.16 and earlier this was slightly
inconsistent as simple GET and POST calls still made use of the
double underscore, where XML and JSON used a single underscore. From
9.17 this has been made consistent so that all requests, and all
responses use a single underscore if prefixes are ON.
Authentication
It is probable that many of the services you are providing will require
that the user authenticate themselves in order for the action to
complete.
As with normal Web apps, the method of authentication
is left largely under your control - you can determine the best approach
that suits your situation. Some approaches include (but are not limited to);
- Pass the Login and Password as fields on all incoming packets.
These fields are then first checked in your code to verify the
request before any other action is taken. The login and password
could be passed as data in the request, or in the HTTP
Authentication header. If using headers then be sure to add code to
your WebHandler procedure in the Authenticate method.
- Allow the user to log in with one request and set the session as
logged in (as you would in a normal web app). Then further requests
from the client can use the SessionID cookie with further requests.
Note however that normal session timeout rules apply here - if no
traffic from the service is received for a pre-determined period of
time, then the session will timeout.
Bear in mind
that because the service will typically be used by some service other
than a browser more complicated authentication schemes are possible.
Basic Authentication
Basic Authentication is preferred over Digest Authentication when used over a TLS connection.
To activate Basic Authentication in your application;
- Go to the WebServer procedure, Extensions, Settings,
Security tab.
Turn on the option Suggest Basic
Authentication for Logged In Pages.
- Go to the WebHandler procedure, to the .Authenticate method.
After the parent call add code to test the pUser
and pPassword parameters. For example;
Access:Users.Open()
Access:Users.UseFile()
User:Login =
pUser
If Access:Users.Fetch(User:Key) = level:Benign
if pPassword = User:Password
ReturnValue =
true
end
End
Access:Users.Close()
NOTE:
The above code is a simplification, Passwords should be stored
as SALTed, HASHed values whenever possible.
While this code is ideal for an API, it is also possible to call .Authenticate from a LoginForm
procedure, thus allowing this code to work for an API or for a Web App.
Digest Authentication
Digest Authentication is preferred to Basic Authentication when used over a non-secure connection. It
does not transmit the password in plain-text, but rather a hash of the password. However it requires the server
to store the actual password, not a hash of the password, which is not ideal. Then again, using any login over
non-secure connections is not ideal. If you are considering Digest authentication rather consider a TLS connection,
and if you have a TLS connection to the server use Basic rather than Digest authentication.
To implement Digest authentication;
- Go to the WebServer procedure, Extensions, Settings,
Security tab.
Turn on the option Suggest Digest
Authentication for Logged In Pages.
- Go to the WebHandler procedure, to the
.GetPassword method. After the parent call add code to
fetch, and return, the users password, based on the pUser
parameter. For example;
Access:Users.Open()
Access:Users.UseFile()
User:Login =
pUser
If Access:Users.Fetch(User:Key) = level:Benign
ReturnValue
= User:Password
end
End
Access:Users.Close()
Sessions
Authenticating a user on each request can be expensive. A SessionID is one way of "remembering" the user
and "knowing" they are already logged in. NetTalk automatically
creates sessions for all connections, and if the client makes use of
the Session then (even if they pass authentication information) then
the authentication code is not re-run.
Session ID's are
included, as a SetCookie header, in every result returned by the
server. If a client respects this cookie setting, and returns the
cookie with each request, then no authentication is required, and
the response will hence be faster. For this reason it is recommended
that clients respect the cookie field and return the cookie.
[End of this document]