Working With REST in Oracle JET

In this final article in a series sharing what I've learned about Oracle JET, I wanted to look into the Oracle JET Common Model & Collection API, a client-side JavaScript API for accessing remote REST web services and plugging them into our JET UI components.

A full list of Oracle JET articles to date are available at the end of this article.

Before diving into the Common Model & Collection API, I thought it might be useful to explain the changing web architectures over the years, which will hopefully give context of why the Common Model & Collection API is useful.

Do I Look Fat in This Web Client?

Back in the good old days of web programming we hand-crafted the HTML, and the most controversial concept at the time was the inclusion of animated gifs. Ultimately the HTML stored in a file on a server was fetched and rendered in a browser. Besides a bit of a storage on the server, and a bit of memory in the browser, web pages being <1k meant everything was thin and lean.

But programmers being what programmers are, we decided this simply wasn't good enough.  We needed more challenges, we wanted to create dynamic HTML with dynamic content. (Sigh, if only we had realized where this led to! ;-)  As JavaScript was in its infancy, we looked to server side application servers to provide us languages like PHP and Java to give us some real grunt in producing content procedurally.

Thus was born the fat server (or more politically correct thick server ;-)

As part of its duties the fat server would generate HTML on the fly based on server-side logic, and as part of that tasks could call out to remote services to retrieve content to include in the HTML.  The concept of REST was in its infancy, so likely this was to databases and media storage, but point being the server did all this work. 

Meanwhile, from the client's perspective the browser still received HTML, CSS, and fairly minimal JavaScript, overall it was happy doing what it had always done, displaying the received content to users.

But programmers being what programmers are, we realized we'd dug a hole for ourselves.  As our websites became successful and more remote users piled on, the server resources required to serve all the content meant we had to build a server the side of the Queen Mary, or, building complex distributed systems and load balanced solutions, and then we started getting into issues of maintaining state across servers during a high availability failover.  Sigh, we realized, thick server architectures might not be the end-all solution to everything (though they still have their uses certainly).

Meanwhile, in the browser world, HTML, CSS & JavaScript continued to evolve and improve, to the point that much of the work we did on the server, could now be offloaded to the client instead.

Thus was born the rich client.

Rich clients take advantage of offloading the resource requirements onto the 1000s of web clients rather than placing this requirement on the finitely resourced web servers.  However with much of the application logic passed off to the client, it now became the client's responsibility to access any remote services required for the HTML.

In that context, AJAX came along to provide browsers the ability to fetch remote HTTP content asynchronously to the main web page which was ideal. In turn, REST HTTP web services became a predominant way of sharing data for web & mobile applications on the internet. A golden age of web development was upon us again.

Of course, working with remote REST web services is rather tedious in JavaScript, many remote REST web services are no better than CRUD (create-read-update-delete) APIs, and writing the boilerplate code to access these is very repetitive. In turn working with web component frameworks, you need to plug the remote REST data into your components, yet another somewhat tedious task.

And thus was born, the Oracle JET Common Model & Collection API.

Oracle JET Common Model & Collection API

So what is the Common Model & Collection API?  Ultimately it's a set of JavaScript APIs bundled with Oracle JET, which allows you to access RESTful remote JSON services. Under the covers, it makes use of AJAX to call the remote services over HTTP transparently to the web developer.

Of advantage to Oracle JET and the web frameworks it supports, the API's result can be published as Knockout observables and we can bind these to our Oracle JET web UI components.

It's worth saying for those familiar with it, the JET API is based and compliant with the BackboneJS model, collection & events model, though it explicitly excludes some functionality such as the Backbone views.

Alright, let's dig deeper to see what it can provide us.

Consider the following REST endpoints for serving HR employee related services:

Given we're familiar with REST services, we know that GET /hr/employees returns a collection of employees, while the remaining methods provide services such as read, update and delete on a single employee model object in the collection.

With this in mind say our JET application wants to access these services, potentially the ability to create, update and delete employees from our web app remotely, and also fetch the collection of employees or one employee identified by their employeeID.

For our JET app to interface with these REST services, we introduce the oj.Model and oj.Collection objects into our view model layer:

define([.etc.,'ojs/ojmodel', 'ojs/ojtable', 'ojs/ojcollectiontabledatasource'],
 function(.etc.) {
  function MyViewModel() {
   var self = this;

   self.EmpDef = oj.Model.extend({
    url: "http://server/hr/employees",
    idAttribute: "employeeId"
   });

   self.EmpsDef = oj.Collection.extend({
    url: "http://server/hr/employees",
    model: new self.EmpDef,
    comparator: "employeeId"
   });
   var employees = new self.EmpsDef;
   ..etc..

In order to use oj.Model & oj.Collection, we import ojs/ojmodel into our RequireJS dependencies.  Take note of ojs/ojtable and ojs/ojcollectiontabledatasource added too, we'll mention those more in a minute.

From there we construct an instance of oj.Model by calling extend(), which will represent exactly one model object in the remote service, an employee in our case.  In the very simple example above we then define the base URL of the service, in this article  fictitious http://server/hr/employees URL.  In the returned payload of employees we must also identify what is the primary key of the JSON payload, for our example employeeId.

Having constructed the oj.Model, we then construct an instance of oj.Collection via extend(), which will represent the collection of employees, our model objects.  In doing so we have to also supply the base URL of our collection: http://server/hr/employees.  We then reference the model object we created so the collection knows what model objects it's individual comprised of, and finally we identify a property in the JSON payload that can be used to compare one model object from another ... for our purposes that is employeeId ... which is used for sorting.

Having constructed the oj.Model & oj.Collection objects, we might like to plug the data into an Oracle JET ojTable.  As example our table HTML could be as follows:

<table id="empsTable"
 data-bind="ojComponent: {component: 'ojTable', data: datasource, 
   columns: [{headerText:'ID',field:'employeeId'}, {headerText:'Name',field:'name'}],
   rowTemplate: 'rowScript'}">
</table>
<script id="rowScript "type="text/html">
 <tr>
  <td data-bind="text: employeeId"></td>
  <td data-bind="text: name"></td>
 </tr>
</script>

Specifically note the ojTable's "data: datasource" mapping.  As per the previous article, the ojTable is a collection object requiring a data binding of type oj.TableDataSource.  From the oj.Model & oj.Collection perspective we must provide it an instance of this as a Knockout observable binding.  As such in our earlier JavaScript we need to extend the example as follows:

self.employeesObservable = ko.observable(employees);
self.datasource = new oj.CollectionTableDataSource(self.employeesObservable());

oj.CollectionTableDataSource as a type of oj.TableDataSource provides the ability to map our oj.Model & oj.Collection against the data structures required by ojTable.

Voila! For a simple table, there is no more code required, from here the API will take care of fetching from the remote REST web service for us to populate the table when rendered automatically for us.

The Curse of Cross-Origin HTTP Requests

In developing our rich client web applications, as we've described the HTML/CSS/JS resources served via one domain may end up accessing services such as REST from one or more other domains:

However at runtime in the browser for your app you're likely to see an error like this in the browser console when the app attempts to access the services in the other domains:

This error is thanks to the enforcement of cross-origin HTTP requests protection in all modern browsers, often referred to as CORS (though I believe strictly speaking CORS is the solution, not the problem, more on this in a moment). A good summary of what this is comes from MDN:

A resource makes a cross-origin HTTP request when it requests a resource from a different domain than the one which the first resource itself serves. For example, an HTML page served from http://domain-a.com makes an <img> src request for http://domain-b.com/image.jpg. Many pages on the web today load resources like CSS stylesheets, images and scripts from separate domains.

For security reasons, browsers restrict cross-origin HTTP requests initiated from within scripts. For example, XMLHttpRequest and Fetch follow the same-origin policy. So, a web application using XMLHttpRequest or Fetch could only make HTTP requests to its own domain. To improve web applications, developers asked browser vendors to allow cross-domain requests.

As described these restrictions are enforced by modern browsers, baked into a W3C standard.

As part of that standard is the solution we referred earlier, CORS, an abbreviation for Cross-Origin Resource Sharing.  CORS essentially allows the remote domain services to state as a HTTP header, what other domains they are happy to share their services with:

This always feels a little counterintuitive when you come to implement it, as it is the browser running the web application that throws and enforces the error.  Shouldn't it be the web server serving the web content that states the domain exemption for the remote service? 

Be it as it may, your web application is attempting to access services which may have not been built by you.  So, in essence, you need privileges on that remote service, so that remote service needs to grant them to you.  And it is browser enforces this policy even though its rendering your web page.

The mechanics of how this works is the remote domain need to define via a HTTP header Access-Control-Allow-Origin what other domains it will trust, namely your JET application's domain.  This header can include a comma delimited list of domains, hardcoded or wildcoded domains such as http://acme.com or http://*.acme.com.

In order for the browser to determine does it have privileges to access the remote domain, before accessing the remote services via HTTP GET/PUT/POST/DELETE, it will make what's called a "pre-flight" request to the remote domain, essentially a HTTP OPTION call.  A HTTP OPTION call is much like a HTTP HEAD call where no data/body is fetched, no operation is undertaken, but the server has a chance to return the HTTP headers for that service, which includes the Access-Control-Origin header mentioned earlier.  This tells the browser which other domains the remote service trusts.  In receiving the list of trusted domains in that header, the browser will compare this to where the web content is being served, and if it does come from a trusted domain it'll then continue with the normal HTTP calls to the remote service that if first intended to use.  If the domain is not in the trust list, then we get the error we saw earlier in the browser:

Ultimately during development CORS is a bit of a pain as it often stops quick playing and testing of new code, but, is something you need to understand certainly by the time your applications gets to production.

Does CORS Affect Hybrid-Mobile Apps?

It's worth concluding this discussion on CORS with consideration of hybrid-mobile apps.  As we know Oracle JET builds both web and hybrid-mobile apps.  As we just described for a JET app running in a web browser, the browser will enforce CORS regardless, so you need to know how to solve this.

What about hybrid-mobile apps running in Cordova on a mobile device?  As we know, Cordova apps when deployed to Android and iOS, the app runs in WebView and UIWebView respectively.    Is CORS enforced?

The simple answer is no.  However there is the caveat if you alternatively decide to run your Oracle JET+Cordova application via the desktop browser using "grunt server --browser",  CORS *is* enforced as you are still running the app in a browser.

Working Programmatically

Returning to our discussion the Common Model & Collection API, as we will to interact with the remote API and data in detail in our application, it's useful to have an understanding of what programmatic APIs are available rather than just wiring the API straight into our UI components.

Given we previously wired up an oj.Model & oj.Collection objects to work with our remote HR employees service:

self.EmpDef = oj.Model.extend({
  url: "http://server/hr/employees",
  idAttribute: "employeeId"
});

self.EmpsDef = oj.Collection.extend({
  url: "http://server/hr/employees",
  model: new self.EmpDef,
  comparator: "employeeId"
});
var employees = new self.EmpsDef;

The following code shows how we would programmatically fetch a single employee by their employeeId of 3:

var findEmp = new self.EmpModelDef({employeeId:3}, employees);

var emp = findEmp.fetch({ 
            success: function(model, response, options) {    
               // model = fetched object  
               console.log("employee.fetch success");    
            },      
               error: function(model, xhr, options) {
                 console.log("employee.fetch error");      
               }});

Alternatively, if we wanted to fetch all employees from the collection we use:

employees.fetch({
  success: function(collection, response, options) {      
    // collection = fetched objects
    console.log("employees.fetch success");
  },      
  error: function(collection, xhr, options) {      
     console.log("departments.fetch error");      
  }});

To create a new employee, and then save it to remote service we call:

var newEmp = new self.EmpModelDef({name:"Bob Smith"}, employees);

newEmp.save(null, {
  success: function(model, response, options) {                       
    console.log("employee.save success");
  },
  error: function(model, xhr, options) {                     
    console.log("employee.save error");                   
  }});

It is not necessary to supply all the properties required for the object, however, the remote server may complain if you do so, so you need to be mindful of what it requires at save time.  It's worth also noting in that last example it is assumed the client is not responsible for assigning the employeeId, rather, the server will allocate one in the response.

Deleting an employee is also fairly trivial:

self.deleteEmp = new self.EmpModelDef({employeeId:36}, employees);

self.deleteEmp.destroy({
  success: function(model, response, options) {                     
    console.log("department.delete success");
  },
  error: function(model, xhr, options) {                       
    console.log("department.delete error");
  }});

What Does a Compliant REST Web Service Look Like?

If you've worked with REST web services enough, you will know that there is really no standards for what the various endpoints could like.  From Oracle's perspective we tend to view REST services as CRUD operations as per this image you've seen earlier:

....but in my personal experience I've seen REST services where absolutely all the operations are supported through a single POST call.  Arguably you may say that's not REST, but when you don't own the remote web service you have to access in your JET application, and your told to get the job done, academic discussions go out the window.

So this raises naturally two questions.  First for the default behaviour of the Oracle JET Common Model & Collection API, what does it expect the REST services to look like?  And second, if the remote REST services don't comply with what the API requires, what can we do about this?

Let's address the first question.

Building Compliant REST Endpoints for the Common Model & Collection API

So let's say we were building an Oracle JET application where we were also in charge of building the REST endpoints for the data the JET application will consume via the Common Model & Collection API.  (I hope we're getting paid well).  What would those endpoints need to look like?

As per the diagram above, if the service was for working with HR employees data, we would expect a collection GET endpoint to return all employees, a GET endpoint to return a specific employee identified by an ID, a POST endpoint to create new employees, a PUT to update an employee, and a DELETE to remove an employee.

Specifically for the GET collection endpoint a GET request would include no payloads, and the server response would carry a JSON array of objects:

Alternatively, for fetching a single employee, we would provide a GET object endpoint which includes the ID of the employee we want to fetch in the URL.  Again the request payload would be empty, but the response would carry the JSON object representing the employee fetched:

What happens if the employee fetched doesn't exist?  We are free to return a 404 HTTP error code, but we can also more information in the payload as follows:

To create employees we provide a POST and as part of the request payload we pass the JSON object representing the new employee.  Note the client does not have to pass in the object ID, employeeId, as in this case it is assumed the server will generate the ID for us.  In saying this for a successful 201 response from the server we can return a success payload including the status and the new employeeId created by the server:

In order to update an employee on the server we can provide either a PUT and/or PATCH endpoint.  The request object carries the updated employee, the response carries the success.

The Common Model & Collection API's default behavior will use PUT calls sending the entire JSON object regardless if all the properties have been updated or not.  If you want to use a more efficient PATCH call from the client where only the changed attributes are sent to the server, the oj.Model.save method includes a patch boolean to control this.

To complete the set, a DELETE operation would like the following:

Working With Messy REST Web Services

In the previous section we discussed the ideal, where-in building an Oracle JET application, you also own the REST web services the application is going to use, so you can build them to the specification required by the Common Model & Collection API.

Let's now start considering the worst case, where you don't own the REST web services, and they are not compliant?  What could you have to face? And how does the Common Model & Collection API solve this?

The first issue you may hit is while the remote REST web service might have fairly standard CRUD endpoints, that is GET/POST/PUT/DELETE, it could be the JSON payloads for incoming and outgoing data is complex and messy.

The API provides the solution to this for providing the ability to hook in functions that handle the reading of JSON payload from the remote web service, where you can undertake a transformation of the data for your needs, and also the opposite when saving.

Consider the following example:

// Transform incoming payloads
parseEmp = function(resp) {
 return { 
   employeeId: resp.empId,
   name: resp.name,
   address: resp.address.street + " " + resp.address.suburb
  ..etc..
 };
};

// Transform outgoing payloads
empSave = function(record) {
  return {
    empId: record.employeeId,
    ..etc..
  };
};

self.EmpModelDef = oj.Model.extend({
  url: "http://server/hr/employees",
  parse: parseEmp,
  parseSave: empSave,
  idAttribute: "empId"
});

In the oj.Model.extend() call it supports the ability to define parse and parseSave functions.  In the parse function called when data is fetched from the remote service, called paresEmp, has access to the 'response' object from the remote service.  The parseEmp function is built to return the payload as we require it.  In this example I've mapped the remote services empId property to employeeId for our needs, name hasn't changed, and created a concatenated address property out of the other address fields.

Conversely, the function empSave set for the parseSave property on the oj.Model.extend() call, is used when we create/update/delete an employee object.  Within the empSave function it receives the 'record', that is the outgoing object we need to transform into the format required by the remote service, returned from the function.  In the example empSave function you can see an attempt to convert our employeeId back into empId for the remote service.

This solves the problem of messy payloads.  What about more complex situations where the URLs of the REST endpoints don't fit our expectations?  Maybe, for example, the DELETE operation doesn't want the URL pattern /hr/employees/1234, but rather something like /hr/delete/employee=1234?

For this the Common Model & Collection API allows us to completely override the URL operations of the collection.  So when we define your oj.Model & oj.Collection as follows:

self.EmpModelDef = oj.Model.extend({
  url: "http://server/hr/employees",
  idAttribute: "employeeId"
});

self.EmpColDef = oj.Collection.extend({
  customUrl: getUrl,
  model: new self.EmpModelDef,
  comparator: "employeeId"
});

...note the use of the customUrl property and associated getUrl function used on the oj.Collection.  The following code shows what we can do in the associated function:

function getUrl(operation, collection, options) {
  var request = {};
  if (operation === "create") {
    return null; // Use default operation for create
  } else if (operation === "delete") {
    return "http://server/destroy/model=' + options["recordID"];
  } else if (operation === "read") {
    request["url"] = "http://server/fetch";
    if (options["sort"]) {
      request["url"] += "/order=" + options["sort"] + ";" + options["sortDir"];
    }
    request["headers"] = {myCustomHeader: "header-value"};   
    request["mimeType"] = "text/plain";
  } else { // update or patch
    request["url"] = "http://update/model=" + options["recordID"]; 
    request["type"] = "POST";
    request["beforeSend"] = myBeforeSendCallback;
  }
  return request;
}

As you can see we can totally override the URLs required, include HTTP headers, and even include callbacks.  There's a lot to cover in that little example and I think too much for this article, but it gives you a test of what's possible.

The previous example is of course just manipulating the Common Model & Collection API HTTP request, but only REST URL and HTTP headers.  What if we needed to totally override the framework, including URL, headers, payload, everything, to say support calling a 3rd party SDK which undertakes all the remote calls instead?  This is where the sync property on oj.Collection comes in:

self.EmpModelDef = oj.Model.extend({
  // url not required when using sync
  idAttribute: "employeeId"
});

self.EmpColDef = oj.Collection.extend({
  // url not required when using sync
  model: new self.EmpModelDef,
  comparator: "employeeId",
  sync: empSync
});

As you can see in the definition of oj.Model & oj.Collection above we don't define a URL.  Rather in the oj. Collection object we are defining a call to an empSync function via the sync property.

The empSync function looks as follows:

function empSync(method, model, options) {        
  var httpUrl = "hr/employees";        
  var httpVerb = "UNKNOWN";        
  var httpPayload = null;        

  if (method === "read") {
    httpUrl = (model instanceof oj.Model) ? httpUrl + "/" + model.id : httpUrl;
    httpVerb = "GET";
    httpPayload = null;
  } else if (method === "create") {
    // httpUrl – POST doesn't require ID in URL
    httpVerb = "POST";
    httpPayload = model.toJSON();
  } else { //patch/update/delete
    ..etc..
  }
  // Example of embedded Oracle Mobile Cloud Service SDK to handle REST calls
  mcsconfig.MobileBackend.CustomCode.invokeCustomCodeJSONRequest(httpUrl, httpVerb, httpPayload,            
    function(statusCode, data, headers) {           
      options["success"](data, null, options);          
    },
    function(statusCode, data) {
      options["error"](data, null, options);
    });
};

In the above empSync function the end goal is to call a 3rd party SDK function to do the fetching of data for us (for the Oracle MCS readers among us, the call to mcsconfig.MobileBackend.CustomCode.invokeCustomCodeJSONRequest() , that function is part of the MCS JavaScript and Cordova SDKs for calling the MCS Custom APIs).

That function has its own requirements about the URL to call, REST verb and payload, so we are using the empSync function here to determine what operation the Common Model & Collection API is undertaking, then transforming those operations into what the remote SDK requires.

Again going into this in more detail is possibly overkill for this article, but I hope I've opened your eyes to the opportunities by exploring this option.  Obviously if I can override the complete set of HTTP calls the Common Model & Collection API makes by using the sync property, that means I could for example not use HTTP at all, and alternatively could use websockets for example!

Conclusion

As we've seen  the Common Model & Collection API is fairly easy to use when the REST endpoints are fairly vanilla and map to what it expects.  In the case where the REST endpoints are messy, either messy payloads, messy URLs and headers and so on, luckily the Common Model & Collection API is very adaptable to overriding it's default behaviour so we can work with the challenging web services out there.

Article Series Conclusion

At this point, I've achieved what I set out to do with this article series. Personally, I wanted to explore the basics of Oracle JET, from installation, to its dependencies such as RequireJS & KnockoutJS, to building simple pages of JET components and laying the pages out, to slightly more complex areas of single page applications and finally here the REST support.

In time, I may explore more complex areas, but, personally, I wanted to start with a good grounding, learn the basics, then apply that knowledge to the more advanced use cases. Personally, I find it's often easier to establish a solid base rather than diving in. YMMV.

My other main goal is while learning the basics, I was hoping to share what I've learned, in the manner that I've learned it. As I often say when telling people how to create a training course, when teaching the A-B-Cs of something, the order of the A-B-Cs is as important as the letters themselves. In other words, the path to learning (how I built my knowledge up) is as valuable to you as the pure technical information we've covered.

This is a long was of saying, I hope you found this article series useful as I did in learning Oracle JET.

Happy JET-setting ;-)!

Article Series Links

The complete series of articles published to date can be found here:

• Working with RequireJS in Oracle JET

• Investigating KnockoutJS in Oracle JET

• Working With Oracle JET UI Web Components

• Responsive Web Design With Oracle JET

• Single Page Applications in Oracle JET