When Localstorage fails — a tale of cookies and crumbs

Posted by Amanda DaSilva on

Here at Grubhub, we use localstorage to persist state in our applications. We migrated our main consumer-facing application to Angular some time ago, and it has since been a cheap, easy option to persist state. The problem? Safari private mode. Safari private mode sets the localstorage quota to zero, which triggers a javascript exception on each call to localstorage.setItem and prevents applications from storing data. For us, this meant that a group of users were unable use our site.

Quick Band Aid

Our temporary solution was to wrap localstorage calls with try/catch blocks. When these blocks intercepted a javascript error, the application redirected users to a page explaining that they could not use our website in private mode. This was a quick fix for us, thanks to our decision to wrap all localstorage calls in a localStorageUtility class. However, this was not a solution to the problem since it fully prevented Safari private users from interacting with our site.

Cookie Solution

After remedying application crashes with error handling, we came up with a solution to enable  users to order food without localstorage. The plan: optimize and reduce localstorage usage and store this data in cookies when localstorage is not available.

Disclaimer: We went about this task knowing that it is not advisable to bloat application cookies since it will slow down network requests. However, the architecture of our application made this developer sin permissible. Since we host the Grubhub apis on a separate domain, the only requests our cookies affect are around page load. In our opinion, a slower page load is preferable to a disabled website. If this cookie hack did affect our api requests, we might have come to a different conclusion.

Cookie Crumbs

A cookie-based storage solution requires addressing two main limitations to storing application data in cookies:

  • Individual cookies have a size limit that can vary between browsers.
  • Domains have a cookie limit that can also vary between browsers.

The effect of these limitations is that a cookie storage solution cannot simply set data on a cookie, it needs to handle cases where the data exceeds the cookie size limit. Our response to this is the aptly named cookie crumb. We define a cookie crumb as a cookie fragment that contains some part of a cookie’s data. A crumbled cookie’s value is derived by combining all of its crumbs.

Our crumb solution works because we have a naming convention that allows reassembling the crumbs in the correct order by sorting the keys. We create a crumb by taking either the first or second half of the parent cookie value and deriving a crumb key. Our algorithm appends these crumb keys with a binary value based on whether they are the first or second half of the value. This is a recursive pattern that results in the number of crumbs being a power of two.

This solution addresses the cookie size limit, which has proven to be the main limiting factor for our application. In the event that the application stores the maximum number of cookies or exceeds its data allotment, our implementation of this solution falls back on directing users to a page encouraging them to switch out of private mode. If a limitation to the number of cookies allowed were to become an issue, an algorithm that maximizes the size of each crumb should replace this one.

Implementation Details

For those interested in the details of how we implemented cookie crumbs, below is the relevant code and steps we took:

  • Setup localStorageUtility to handle multiple generic storage solutions. This structure allows the application to easily switch between localstorage, cookie storage, and any future storage implementations of IStorageInterface. We did this in three steps:
    • Define a storage interface (IStorageInterface)
    • Implement the storage interface in a localStorageImpl class
    • Replace localStorage interactions in localStorageUtility with an instance of localStorageImpl.

Our code:


interface IStorageInterface {
		get: (key: string) => any;
		set: (key: string, value: any) => void;
		reset: (prefix: string) => void;
		remove: (key: string) => void;
		dictionary: () => Object;
  • Implement the CookieStorageImpl class. For quick access, this class maintains a dictionary of stored cookies and crumb keys.
    • Constructor: first loads all the browser cookies into a dictionary then calls gatherCrumbs on that dictionary. The first loop inside gatherCrumbs identifies keys that have crumbs – their values are set to the crumbKey value. For each of those keys, a second loop searches the dictionary for keys that are crumbs of the parent key and appends the keys to a list. The final list is sorted, the strings appended, and the parent key is updated with the final encoded value.
    • get(): returns the decoded value for the key in the dictionary.
    • set(param): first calls remove for the key in order to clean up any crumbs that may have been set previously. It then adds the encoded value to the dictionary and tests that the cookie was correctly set. If the cookie was not set correctly, it means the value is too large. Our recursive cookie logic follows:
      • Set the original key to crumbKey so we can quickly identify it as a “crumbled” cookie.
      • Divide the value in half and derive the two new crumb keys. The value crumbKey lets us quickly identify a crumb and adding the numerical suffix allows for sorting the crumbs when reassembling. Call setCrumbs with the new keys and split values.
        • First half key = key + ‘-‘ + crumbKey + ‘-0’
        • Second half key = key + ‘-‘ +crumbKey + ‘-1’
      • setCrumbs will attempt to set the cookie and the dictionary, and once again test that the set cookie is correct. If it is not correct, we recursively call setCrumbs with new keys and the split values.
        • First half key = key + ‘0’
        • Second half key = key + ‘-1’
      • If the length of the value is less than 500, then it is likely that the browser is out of cookie storage and we reset the cookies and redirect to an error.
      • If setting a large cookie is successful, the browser cookies may include:
        • cookieKey-crumbKey-00
        • cookieKey-crumbKey-01
        • cookieKey-crumbKey-10
        • cookieKey-crumbKey-11
    • remove(key): deletes the key from the dictionary and expires the cookie. Then, the call looks for any crumb references to delete in the same way.

Our code


export class CookieStorageImpl extends StorageConstants implements IStorageInterface{
    private loadedCookies;
    private crumbKey = 'ngCrumb';
    private defaultExp = 30;
    constructor(private $browserCookieUtils: BrowserCookies, private $location) {
        super();
        this.init();
    }
    public get(key: string): any {
        if (this.loadedCookies[key]) {
            return this.decode(this.loadedCookies[key]);
        }
    }
    public set(key: string, value: any, expDays?: number) {
        let encodedVal = this.encode(value);
        this.remove(key);
        this.loadedCookies[key] = encodedVal;
        expDays = angular.isDefined(expDays) ? expDays : defaultExp;
        this.$browserCookieUtils.createCookie(key, encodedVal, expDays, true, true);

        if (this.cookieFailure(key, encodedVal)) {
            // if this fails, the cookie is likely too large, and we need to divide the cookie into crumbs
            this.$browserCookieUtils.createCookie(key, this.encode(this.crumbKey), expDays, true, true);
            this.setCrumbs(key + '-' + this.crumbKey + '-0', value.slice(0, value.length / 2), expDays);
            this.setCrumbs(key + '-' + this.crumbKey + '-1', value.slice(value.length / 2), expDays);           
        }
    }
    public remove(key: string) {
        delete this.loadedCookies[key];
        this.$browserCookieUtils.deleteCookie(key);

        // remove crumbs
        for (let crumb in this.loadedCookies) {
            if (crumb.indexOf(key + '-' + this.crumbKey) > -1) {
                delete this.loadedCookies[crumb];
                this.$browserCookieUtils.deleteCookie(crumb);
            }
        }
    }
    public reset(prefix: string): boolean {
        let hasReset= false;
        for (let key in this.loadedCookies) {
            if (key.substring(0, prefix.length) === prefix) {
                this.remove(key);
                hasReset = true;
            }
        }
        return hasReset;
    }
    public dictionary(): Object {
        return this.loadedCookies;
    }

    private init() {
       let cookies = document.cookie,
            tempStore = {},
            temp;
        // loading cookies
        cookies.split(';').forEach((value) => {
            temp = value.split('=');
            if (temp[0] && temp[1]) {
                tempStore[temp[0].trim()] = temp[1].trim();
            }
        });
        this.loadedCookies = this.gatherCrumbs(tempStore);
    }

/**
 * helper function to re-assemble cookies that have been 'crumbled' to fit
 */
    private gatherCrumbs(cookieStore: Object): any {
        for (let key in cookieStore) {
            if (key.indexOf(this.crumbKey) === -1 && cookieStore[key] === this.encode(this.crumbKey)) {
                let crumbs = [];
                 for (let possibleCrumb in cookieStore) {
                     if (possibleCrumb.indexOf(key + '-') > -1) {
                         crumbs.push(possibleCrumb);
                     }
                 }
                 crumbs.sort();
                 cookieStore[key] = crumbs.reduce((previousValue, currentKey) => {
                     let newVal = previousValue + cookieStore[currentKey];
                     cookieStore[currentKey] = '';
                     return newVal;
                 }, '');
            }
        }
        return cookieStore;
    }

/**
 * helper function for setting and dividing cookies until the fit. We assume that
 * all modern browsers will store at least 500 characters in a cookie.
 */
    private setCrumbs(key: string, value: string, expDays: number): void {
        this.$browserCookieUtils.createCookie(key, value, expDays, true, true);

        if (value && value.length > 500 && this.cookieFailure(key, value)) { // need to split into spaller fragments
            this.remove(key);
            this.setCrumbs(key + '0', value.slice(0, value.length / 2), expDays);
            this.setCrumbs(key + '1', value.slice(value.length / 2), expDays);
        } else if (value && value.length <= 500) { // ran out of cookie storage or cookies disabled, clean up and redirect
            this.reset(this.PREFIX_KEY);
            this.$location.path('/cookiesError');
        } else { // success
            this.loadedCookies[key] = '';
        }
    }
    private cookieFailure(key: string, value: string): boolean {
        return this.$browserCookieUtils.getCookie(key).length !== value.length;
    }

    private encode(value: any): string {
        return encodeURIComponent(angular.toJson(value));
    }
    private decode(value: any): any {
        if (value === '' || !angular.isDefined(value)) {
            return;
        }
        return angular.fromJson(decodeURIComponent(value));
    }
}
	

Storage Today

This cookie solution worked for our diner app temporarily, until the total size of our cookie data began exceeding browser limitations (iOS < 10, this limit is about 8kb). We have little control over this since our production environment includes cookies from 3rd party sources that can fluctuate in size and significantly contribute to cookie bloat. Today we have implemented a server-side storage solution to serve our users without access to localstorage.

Although we no longer use it, we believe that this cookie solution may be helpful to others. For us, it successfully bought some time to implement server side storage. For others less reliant on localstorage and with a separate api domain, it could prove to be the entire solution for a frustrating edge case.