Configure the Job Function

Next, we’ll begin coding the news application by configuring the job function component.

The function’s directory, in this case functions/NewsApp, contains:

• The main.py main function file
• The catalyst-config.json configuration file
• requirements.txt file

We will be adding code in the main.py file. You will not be modifying the code of the configuration and dependency files.

As mentioned in the introduction, the job function performs two tasks: making the API calls to the NewsAPI to fetch news, populating the news in Catalyst Data Store. The API calls are made using an API key provided by NewsAPI.

Register with NewsAPI

Before you code the job function, you must register for a free developer subscription with NewsAPI and obtain the API key in the following way:

1. Visit https://newsapi.org/register.

2. Provide the required details and click Submit.

After your registration is complete, NewsAPI will provide you with an API key. You must use this in your cron function, as instructed after the code section.

Add Function Code

You can copy the code below and paste it in main.py located in the functions/NewsApp directory of your project and save the file. You can use any IDE to work with the application’s files.

    

        
main.py
        

            
copy
            

                

            
        
    
    

    

         import logging
import zcatalyst_sdk
import requests

logger = logging.getLogger()

def handler(job_request, context):
    logger.info("Fetching news!")
    # The names of the tables in the Data Store where the news items need to be stored
    table_name = ["HEADLINES", "BUSINESS", "ENTERTAINMENT", "HEALTH", "SCIENCE", "SPORTS", "TECHNOLOGY"]
    country = "US" #Fetches the news items from the United States of America
    api_key = "cc30ddcaxxxxxxxxxx" # Provide the API key you obtained from NewsAPI inside the quotations
    fetch_url = "https://newsapi.org/v2/top-headlines"
    
    #Fetches the breaking news of different categories with their headlines
    for category in table_name:
        
        # Constructing required to fetch news
        params = {
            'country':country,
            'apiKey':api_key,
        }
        if category != "HEADLINES":
            params['category'] = category
        # Makes an API call to the News API to fetch the data
        response = requests.get(fetch_url, params=params)
        # If the response is 200, the data is moved to the Data Store. If not, the error is logged.
        if response.status_code == 200:
            data = response.json()
            try:
                if 'articles' in data:
                    array_data = data['articles'][:15]
                    logger.info("15 days "+str(array_data))
                    # This method inserts/updates the news in the Data Store
                    push_to_datastore(array_data,category)
            except requests.RequestException as e:
                logger.error("Request failed: "+ str(e))
                context.close_with_failure() # end of application with failure
            # The actions are logged. You can check the logs from Catalyst Logs.
            logger.info("New updated")
        else:
            logger.error("Request failed with status :: ",response.status_code)
            context.close_with_failure() # end of application with failure
    context.close_with_success() # end of application with success



def push_to_datastore(sliced_data,tableName):
    app = zcatalyst_sdk.initialize()
    zcql_service = app.zcql()
    datastore_service = app.datastore()
    #Defines the ZCQL query that will be used to find out the number of rows in a table
    query = "select ROWID from "+tableName
    zcql_result = zcql_service.execute_query(query)
    table_instance = datastore_service.table(tableName)
    #Inserts / Update the data obtained from the API into datastore
    for i in range(0,14):
        title = sliced_data[i]['title']
        url = sliced_data[i]['url']
        if len(zcql_result) == 0:
            # inserting new row
            row_data= {'title':title,'url':url}
            table_instance.insert_row(row_data)
            logger.info("Row inserted "+ str(row_data))
        else:
            # updating row
            row_data= {'title':title,'url':url,'ROWID':zcql_result[i][tableName]['ROWID']}
            table_instance.update_row(row_data)
            logger.info("Row inserted "+ str(row_data))

         
View more
    

The job function is now configured. We will discuss the application’s architecture after we configure the client.