Recursive async functions

Apps Script (intermediate level) posted on 17th May 2018


Background

I have a GraphQL server on which I've deliberately limited the amount of data that can be returned in a single query to avoid daft requests. That means that you need to do paging to get a set of results of more than that limit. There are various ways of paging in GraphQL, but this one uses  a simple system of limits and offsets to get chunks of results. Something like this
query ($limit: Int , $offset: Int) {
   People ( limit: $limit , offset: $offset ) {
    id
   name
  }
}

That leaves the problem of how exactly to do queries like this when the calls themselves are asynchronous. If they were synchronous, then a simple recursive approach would be easy enough to implement, but asynchronous calls are a little more tricky. Using Promises improves things a little, but using ES6 "await" makes it no more complex than a synchronous solution. Let's take a look. 

Fetcher

Start with a simple fetcher (I'm using axios) that makes a request given a query an some optional variables.
  /**
   * fetcher for graphql
   * @param {string} query the graphql query
   * @param {object} [variables] the optional graphql variables (the .data part of the graphql response)
   * @return {object} the result 
   */
  ns.gfetch = (query, variables) => {
    
    // ns.curl is already set to the endpoint for graphql server
    return axios
      .post(ns.cUrl, {
        query,
        variables
      })
      // strip off the axios wrapper
      .then(result => result.data.data)
      
      .catch(err => {
        // strip down the axios error to something meaningful
        const pack = {
          status: err.response.status,
          statusText: err.response.statusText,
          query,
          variables,
          errors: err.response.data.errors
        };
        // for quick tracking down
        console.log(JSON.stringify(pack));
        return Promise.reject(pack);
      });
  };

Pager

The pager is called to bunch up pages of graphql results into a single response. The inline comments describe what's going on, but in summary
  • Pages are recursively fetched until either there are no more or the max requested is reached.
  • After each page, the offset and limit variables are tweaked to get the next page.
  • The use of await means it reads almost like a synchronous recursion
  ns.paged = (options) => {

    let { vars, q, max, offset, limit, once } = options;
    // total result will be accumulated here
    const bunch = [];

    // set up default values and check params
    offset = offset || 0;
    limit = limit || 50;

    // max is the overall max number of rows to return
    // by default, it's all of them.
    if (max && limit > max) limit = max;

    // once is a single query with no paging required
    if (!once) {
      // if we are paging, we'll need some variables to control it.
      vars = vars || {};
      vars.limit = limit;
      vars.offset = offset;
      // and the query must be set up to allow limit and offet
      if (!q.match("limit:Int")) throw "must have a limit in the query for paging";
      if (!q.match("offset:Int")) throw "must have an offset in the query for paging";
    }

    // this is the recursive function
    // the object returned from graphql can be extracted using its key
    const doFetch = (v) => ns.gfetch(q, v).then(r => {
      // this is the property key of the first object
      const [stem] = Object.keys(r);
      if (Object.keys(r).length !== 1) throw 'paged only handles single top level queries ' + JSON.stringify(Object.keys(r));
      return r[stem];
    });

    // can detect a next page if we havent got enough recs or if the full limit was returned
    const isNextPage = (data) => (bunch.length < max || !max) && data.length === limit && !once;

    // get all the pages
    // its an async function
    const getPages = async function(v) {

      // so we can use await{
      const data = await doFetch(v);

      // accumulate what just got returned
      Array.prototype.push.apply(bunch, data);

      // if there's any more then carry on
      if (isNextPage(data)) {

        // modify the offset to skip what we have
        v.offset = bunch.length + offset;

        // modify the limit if it would return more than we need
        if (max && max - bunch.length < limit) v.limit = max - bunch.length;

        // go round again
        return await getPages(v);
      }
      else {
        return bunch;
      }
    };

    // start the whole thing off and return the accumulated result
    return getPages(vars);


  };

Using it

Get all the people named smith in a single result
  paged({
    q: `query ($value:Value,$limit:Int,$offset:Int) {
          People (value: $value, limit: $limit, offset: $offset) {
            id
            name
          }
        }`,
    vars: {
       value:"Smith"
    }
  }).then (people =>{
    // do something with the result
  });

You want to learn Google Apps Script?

Learning Apps Script, (and transitioning from VBA) are covered comprehensively in my my book, Going Gas - from VBA to Apps script, All formats are available from O'ReillyAmazon and all good bookshops. You can also read a preview on O'Reilly

If you prefer Video style learning I also have two courses available. also published by O'Reilly.
Google Apps Script for Developers and Google Apps Script for Beginners.

Comments