Google Drive as cache

Apps Script (intermediate level) posted on 20th October 2018


If you are after performance and self cleansing then the cacheservice is the best solution for caching, and if you are after permanence and small amounts of data, the properties service is a good solution. You can also get round many of the limitations of each service using my properties and cache plugins, covered in Squeezing more into (and getting more out of) Cache services

However - what if you want 
  • permanence
  • sharing across multiple projects
  • much bigger payloads
At the expense of self expiry and performance you could use my CrusherPluginDriveService

You'll need the cUseful library
1EbLSESpiGkI3PYmJqWh3-rmLkYKAtCNPi1L2YCtMgo2Ut8xMThfJ41Ex

Built in plug-ins

This works exactly like the the plugins for CacheService and PropertyService, and are supported through plug-ins available from the cUseful library. Here's how to use each of them.

Properties Service
  const crusher = new cUseful.CrusherPluginPropertyService().init ({
    store:PropertiesService.getScriptProperties()
  });

Cache Service
  const crusher = new cUseful.CrusherPluginCacheService().init ({
    store:CacheService.getUserCache()
  });

Drive Service
  const crusher = new cUseful.CrusherPluginDriveService().init ({
    store:DriveApp,
    prefix:"/somefolder/toputfilesins"
  });

Methods

as with cacheservice and property service 3 methods are supported. Note that although the expiry dates are respected for whether to return results, the files are not automatically pruned unless accessed as Drive doesnt support automatic expiry pruning so you may need to clean up your folder from time to time.
crusher.put ( key , value [,expiry]);
crusher.get ( key );
crusher.remove (key);

key
This can be of any type, or even an object, for example
crusher.put ( "monday" , value);
crusher.put ( {url:"https://example.com",header:{Authorization:"Bearer ytxxx"}} , value);

value
This can be of any type, for example
crusher.put ("name" , "bruce");
crusher.put ("apiresult", {data:"xyz"}, 20);
crusher.put ("today", new Date());
crusher.put ("myimage",file.getBlob());

Getting the item will reconstitute to its original form. A missing item will return null
const str = crusher.get ("name");
const obj = crusher.get ("apiresult");
const dat = crusher.get ("today");
const blob = crusher.get ("myimage");

An item can be removed like this, irrespective of type.
crusher.remove ("name");

How does it work?

The main techniques are
  • If data is over a certain size, then it will be automatically compressed and uncompressed when retrieved
  • If data is still too large for a given store's limits (which you can set), then it will create a series of linked items, which are reconstituted when retrieved.
  • Objects are stringified and re-parsed automatically when detected
  • Dates are converted to timestamps, then back again when retrieved
  • Blobs are converted to base64, preserving their content type and name, and reconverted when retrieved
  • The store is abstracted from the crusher, so the methods are exactly the same, irrespective of which underlying store is being used.

Options

These examples for the built in property stores and cache stores show some initialization options.

Minimal
You need to at least pass a store to use, and a prefix with the path to an existing Drive folder.

You can set a few other options to affect the behavior, although there's probably not much call for these in normal usage (other than plug-in testing). This example sets small chunk sizes (which would provoke spreading the data over multiple entries) and a small compression threshold (normally compression will actually increase the size of anything under about 200 bytes). By default anything under 250 bytes is not compressed. The default chunksize for the drive plugin is 5mb
  const crusherCacheChunkZip = new cUseful.CrusherPluginDriveService().init ({
    store:DriveApp,
    chunkSize:500000, 
    compressMin:100
  });

Plug-ins

You can write your own plug-ins to support other stores such as databases, cloud storage, even spreadsheets. Essentially - anything that can be used as a key/value store. See Squeezing more into (and getting more out of) Cache services for an example of writing a plugin to support cloud storage.

A benefit of this plugin approach is that each driver is used in exactly the same way. Here's a test that runs 3 different plugins, yet the code is exactly the same, aside from the initialization parameters.

function testCrusher () {
  
  var t = new cUseful.Tester();
  // since we'll be using DriveApp force permission dialog (or add  https://www.googleapis.com/auth/drive  to manifest)
  // DriveApp.getRootFolder()
  
  // generate some random data
  var data = cUseful.getRandomSheetStrings(100,20);
  
  // some  key for digesting
  var key = {random:cUseful.Utils.generateUniqueString() , data:data.slice (0,1).slice (2,2) };
      
  // various chunksizes and drivers
  [{chunkSize:0, store:CacheService.getScriptCache(), driver: "CrusherPluginCacheService", prefix:"na"}, 
   {chunkSize:7777, store:CacheService.getScriptCache(), driver: "CrusherPluginCacheService", prefix:"na"},
   {chunkSize:0, store:PropertiesService.getScriptProperties(), driver: "CrusherPluginPropertyService", prefix:"na"},
   {chunkSize:0, store:DriveApp, driver: "CrusherPluginDriveService", prefix:"/crusher/store/"}
  ].forEach (function (d) {
    
    var cacher = new cUseful[d.driver] ()
    .init ( {
      store: d.store ,
      chunkSize: d.chunkSize,
      prefix: d.prefix
    });
    
    t.describe (d.driver + ' chunksize ' + d.chunkSize , function () {


      t.it ("check write", function() {
        var result = cacher.put ( key , data);
        t.assure ( result>0 , 'written some data');
      });
      
      t.it ("check read", function() {
        var result = cacher.get ( key);
        t.assure ( JSON.stringify(data) === JSON.stringify(result) , 'read should be same as write');
      });
      
      t.it ("check remove" , function () {
        var result = cacher.remove (key);
        t.assure ( result > 0, "failed to remove");
        Logger.log ('>>number of chunks removed ' + result);
      });
      
      t.it ("check removed", function () {
        var result = cacher.get (key);
        t.assure (!result, "item wasnt removed");
      });
    });
    
    
    
  });
}

test results

We can also use this to compare performance  - drive is a lot slower than the others, but of course is a better choice if you need permanence.
  • cache - 500ms
  • property store - 800ms
  • drive - 4500ms

[19-01-27 06:24:30:322 PST] starting section CrusherPluginCacheService chunksize 0
[19-01-27 06:24:30:323 PST] --starting test check write
[19-01-27 06:24:30:542 PST] --ending test check write-OK
[19-01-27 06:24:30:544 PST] --starting test check read
[19-01-27 06:24:30:625 PST] --ending test check read-OK
[19-01-27 06:24:30:626 PST] --starting test check remove
[19-01-27 06:24:30:794 PST] >>number of chunks removed 1
[19-01-27 06:24:30:795 PST] --ending test check remove-OK
[19-01-27 06:24:30:796 PST] --starting test check removed
[19-01-27 06:24:30:829 PST] --ending test check removed-OK
[19-01-27 06:24:30:831 PST] ending section CrusherPluginCacheService chunksize 0
[19-01-27 06:24:30:832 PST] starting section CrusherPluginCacheService chunksize 7777
[19-01-27 06:24:30:833 PST] --starting test check write
[19-01-27 06:24:31:467 PST] --ending test check write-OK
[19-01-27 06:24:31:468 PST] --starting test check read
[19-01-27 06:24:31:675 PST] --ending test check read-OK
[19-01-27 06:24:31:676 PST] --starting test check remove
[19-01-27 06:24:32:173 PST] >>number of chunks removed 5
[19-01-27 06:24:32:174 PST] --ending test check remove-OK
[19-01-27 06:24:32:175 PST] --starting test check removed
[19-01-27 06:24:32:208 PST] --ending test check removed-OK
[19-01-27 06:24:32:209 PST] ending section CrusherPluginCacheService chunksize 7777
[19-01-27 06:24:32:210 PST] starting section CrusherPluginPropertyService chunksize 0
[19-01-27 06:24:32:211 PST] --starting test check write
[19-01-27 06:24:32:586 PST] --ending test check write-OK
[19-01-27 06:24:32:586 PST] --starting test check read
[19-01-27 06:24:32:784 PST] --ending test check read-OK
[19-01-27 06:24:32:785 PST] --starting test check remove
[19-01-27 06:24:32:973 PST] >>number of chunks removed 5
[19-01-27 06:24:32:974 PST] --ending test check remove-OK
[19-01-27 06:24:32:974 PST] --starting test check removed
[19-01-27 06:24:33:003 PST] --ending test check removed-OK
[19-01-27 06:24:33:004 PST] ending section CrusherPluginPropertyService chunksize 0
[19-01-27 06:24:33:734 PST] starting section CrusherPluginDriveService chunksize 0
[19-01-27 06:24:33:735 PST] --starting test check write
[19-01-27 06:24:35:682 PST] --ending test check write-OK
[19-01-27 06:24:35:682 PST] --starting test check read
[19-01-27 06:24:37:433 PST] --ending test check read-OK
[19-01-27 06:24:37:433 PST] --starting test check remove
[19-01-27 06:24:39:024 PST] >>number of chunks removed 1
[19-01-27 06:24:39:024 PST] --ending test check remove-OK
[19-01-27 06:24:39:025 PST] --starting test check removed
[19-01-27 06:24:39:429 PST] --ending test check removed-OK
[19-01-27 06:24:39:430 PST] ending section CrusherPluginDriveService chunksize 0

You want to learn Google Apps Script?

Learning Apps Script, (and transitioning from VBA) are covered comprehensively in my my book, Going Gas - from VBA to Apps script, All formats are available from O'ReillyAmazon and all good bookshops. You can also read a preview on O'Reilly

If you prefer Video style learning I also have two courses available. also published by O'Reilly.
Google Apps Script for Developers and Google Apps Script for Beginners.

Comments