Configure the Job Function

Next, we’ll begin coding the news application by configuring the job function component.

The function’s directory, in this case functions/NewsApp, contains:

• The index.js main function file
• The catalyst-config.json configuration file
• Node modules
• package.json and package-lock.json dependency files

We will be adding code in the index.js file. You will not be modifying the code of the configuration and dependency files.

As mentioned in the introduction, the job function performs two tasks: making the API calls to the NewsAPI to fetch news, populating the news in Catalyst Data Store. The API calls are made using an API key provided by NewsAPI.

Register with NewsAPI

Before you code the job function, you must register for a free developer subscription with NewsAPI and obtain the API key in the following way:

1. Visit https://newsapi.org/register.

2. Provide the required details and click Submit.

After your registration is complete, NewsAPI will provide you with an API key. You must use this in your cron function, as instructed after the code section.

Install Packages for Node.js

The Node.js Cron function requires two packages to be installed: express and axios.

express

We will use the Express framework to manage the routing operations that enable us to fetch and delete files.

To install express, navigate to the function’s directory (functions/NewsApp) in your terminal and execute the following command:

This will install the Express module and save the dependencies.

axios

axios is a promise-based HTTP client that we will use to fetch data by executing the NewsAPI.

To install axios, navigate to the Node function’s directory (functions/NewsApp) and execute the following command:

This will install the module.

You can copy the code below and paste it in index.js located in the functions/NewsApp directory of your project and save the file. You can use any IDE to work with the application’s files.

    

        
index.js
        

            
copy
            

                

            
        
    
    

    

         /**
 * 
 * @param {import("./types/job").JobRequest} jobRequest 
 * @param {import("./types/job").Context} context 
 */

const catalyst = require("zcatalyst-sdk-node");
const axios = require('axios');

const HOST = "https://newsapi.org";
const TABLENAME = [
	"HEADLINES",
	"BUSINESS",
	"ENTERTAINMENT",
	"HEALTH",
	"SCIENCE",
	"SPORTS",
	"TECHNOLOGY",
  ];
  const COUNTRY = "US";
  const APIKEY = "cc30ddcae1684e848f08ed474f6aaf0e";
  const DATASTORE_LOAD = 5;

  const pushNewstoDatastore = async ({ table, rowIds, articles }) => {
	return Promise.all(
	  articles.map(async (article, idx) => {
		const payload = {
		  title: article.title,
		  url: article.url,
		};
		if (rowIds.length === 0) {
		  //insert the new row
		  return table.insertRow(payload);
		}
		return table.updateRow({ ...payload, ROWID: rowIds[idx] });
	  })
	);
  };

module.exports = async (jobRequest, context) => {

	try{
		console.log('Execution started');

		const catalystApp = catalyst.initialize(context);
		const zcqlAPI = catalystApp.zcql();
		const datastoreAPI = catalystApp.datastore();

		
		const metaArr = await Promise.all(
			TABLENAME.map(async (table) => {
			  //construct request path to newsapi
			  let url = `${HOST}/v2/top-headlines?country=${COUNTRY}&apiKey=${APIKEY}`;
			  if (table !== TABLENAME[0]) {
			   url += "&category=" + table;
			  }
			  const response = await axios({
			   method: "GET",
			   url
			 });
		   let data = response.data;
	   
			  //query using zcql to check if row exists
			  const queryResult = await zcqlAPI.executeZCQLQuery(
				`SELECT ROWID FROM ${table}`
			  );
			  return {
				table_name: table,
				table: datastoreAPI.table(table),
				zcql_response: queryResult,
				articles: data.articles.splice(0, 15),
			  };
			})
		  );
		

		  //sync insert/update to datastore
		  for (const meta of metaArr) {
			let rowIds = [];
			while (meta.articles.length > 0) {
			  const chunk = meta.articles.splice(0, DATASTORE_LOAD);
			  if (meta.zcql_response.length > 0) {
				rowIds = meta.zcql_response
				  .splice(0, DATASTORE_LOAD)
				  .map((row) => row[meta.table_name].ROWID);
			  }
			  await pushNewstoDatastore({ ...meta, articles: chunk, rowIds });
			}
			console.log(
			  `${meta.table} table ${
				rowIds.length === 0 ? "inserted" : "updated"
			  } with current news'`
			);
		  }
		context.closeWithSuccess(); //end of application with success
	}catch(err){
		console.log("Error :: "+err);
		context.closeWithFailure(); //end of application with failure
	};
};

         
View more
    

The job function is now configured. We will discuss the application’s architecture after we configure the client.