← TEKNKL :: Blog

Code Anatomy: ... and yes, FlowBoost can be that webhook aggregator!

By Sanford Whiteman Tags: FlowBoost, JS, webhooks, parallelism, code anatomy

In an earlier post I noted how Marketo’s limit of 20 concurrent webhook connections can impact timely Smart Campaign processing.

Instead of running a big batch of leads through multiple independent webhooks, it’s better to run ’em once through a single webhook aggregator ’hook.

That aggregator is simply a webhook-compatible service that connects to other webhook-compatible services in parallel and returns combined results to Marketo. Marketo only calls the aggregator.

While you can build the aggy (just made up that nickname) in any programming language — as the saying goes, “The best language is the one you know” — given a choice, you should use a natively async language like JavaScript[1] because you need to match the scale of the services that are the next hop out.

And, of course, FlowBoost is a JS engine... and not only that, but I’m gonna give you all the code you need in a little bit!

Now, upfront disclaimer: FlowBoost Pro is necessary for this; you can’t do it with the free Community version or even the entry Standard version. That’s because Pro — at least the way we currently have the features laid out — is necessary to make remote connections to non-Marketo servers. But I may change my mind and enable this in Standard if you lobby hard enough.☺

Why such a deep dive into the code anatomy?

“Teach a person to fish” and all that. I could’ve baked this into the FlowBoost engine as something like FBAggregate. But that wouldn’t be any faster to execute (it’s very lightweight code, even if you don’t get it at first) and there wouldn’t be any learnin’ to be had.

Plus, having the code be in “userland” — i.e. right in the Marketo webhook definition — makes fixes and changes instantaneous. The overall philosophy of FlowBoost is to be an open platform for JS-powered flow steps, not locking people into functions that might’ve worked for one customer at one point in time but never evolve.

So away we go...

Our sample services

First, we need a few services to aggregate. I’ll take 2 directly from the earlier post, an Email Verification service and a Firmographic Enrichment service. The first responds like so:

{
  "emailExists": true,
  "mxProvider": "Office 365"
}

The second like so:

{
  "sector": "Manufacturing",
  "size": "500-1000",
  "GSA": true
}

Then I’ll stack on a reCAPTCHA Verification service, to show that this method gets even better when you have 3, 4, or more webhooks to aggregate. The reCAPTCHA responds like so, with true or false of course:

{
  "isHuman": true
}

Starting the FlowBoost/JS code:

Keeping user-managed config at the very top of a FlowBoost webhook is important, just as in a Velocity token or any other “many cooks” type of environment. Strive to get a /* --- NO NEED TO TOUCH BELOW THIS LINE! --- */ comment as high up as possible.

So we’ll start with the only part you’d need to tweak, the list of webhooks, each consisting of a friendly name + URL. For simplicity we assume these are all GET webhooks, not POST, and that they don’t need custom headers — but those configs could be built out quite easily.

It’s a basic JS array-of-objects:

const hooks = [
  { 
	"name" : "emailValidate",
	"url" : "https://www.emailverifysvc.example?email=" + encodeURIComponent({{Lead.Email Address}})
  },
  {
	"name" : "firmEnrich",
	"url" : "https://www.firmographicsvc.example?company=" + encodeURIComponent({{Company.Company Name}})
  },
  {
    "name" : "reCAPTCHA",
    "url" : "https://www.recaptchaverify.example?response=" + encodeURIComponent({{Lead.LastRecaptchaResponse}}) + "&secret=" + encodeURIComponent({{My.RecaptchaSecret}})
  }
];

/* --- NO NEED TO TOUCH BELOW THIS LINE! --- */

Parsing the config

The config block should be easy to understand. It helps later if we have the names in their own array, so let’s pick them out:

const hookNames = hooks.map( hook => hook.name );

Now hookNames is a 1-dimensional array extracted from hooks:

[ "emailValidate", "firmEnrich", "reCAPTCHA" ]

Making the HTTP GETters (or POSTers)

After the Marketo {{lead.tokens}} are interpolated (this happens on-the-fly when Marketo sends the ’hook) the hooks config looks like so:

const hooks = [
  { 
	"name" : "emailValidate",
	"url" : "https://www.emailverifysvc.example?email=sandy%40t%C3%A9k%C3%B1kl.com"
  },
  {
	"name" : "firmEnrich",
	"url" : "https://www.firmographicsvc.example?company=TEKNKL
  },
  {
    "name" : "reCAPTCHA",
    "url" : "https://www.recaptchaverify.example?response=456&secret=123"
  }
];

So we need to turn those url strings into things that can be fetched.

FBHttp.fetch in FlowBoost works exactly like the Fetch API’s fetch method in browsers. (It’s just hanging off an FB*-branded object for... reasons.[2])

So map the set of URLs to corresponding Fetch promises:

/** @repeated
const hookNames = hooks.map( hook => hook.name );
*/
const fetchers = hooks.map( hook => FBHttp.fetch(hook.url) );

Running the requests, parsing the responses

Now, we put the requests on the wire and catch the JSON responses (today’s sample services all return JSON, but we could also support XML if necessary):

/** @repeated
const hookNames = hooks.map( hook => hook.name );
const fetchers = hooks.map( hook => FBHttp.fetch(hook.url) );
*/

Promise.all( fetchers )
.then( responses => responses.map( resp => resp.json() ) )
.then( jsonparsers => Promise.all( jsonparsers ) )

Promise.all continues once all of the fetchers have responded. Since the fetchers run in parallel the total time elapsed will be the time it takes the slowest of the 3 services to respond (the difference may be negligible, of course, which is even better).

Aggregating the responses

Now the rest of the magic happens.

After FBHttp.fetch runs and the webhooks’ JSON responses are parsed, you’ll have an array with the responses, in the same order as in hooks (the config block up top). The responses aren’t tagged with the original name, so the array looks like so:

[{
  "emailExists": true,
  "mxProvider": "Office 365"
},
{
  "sector": "Manufacturing",
  "size": "500-1000",
  "GSA": true
},
{
  "isHuman": true
}]

Looks pretty good, and you could send that response back to Marketo and work with it. But you’d have to manage the original order in the request, as your Response Mappings would have to be indexed: response[0], response[1], etc. That’s cumbersome and prone to error and I wouldn’t dare leave you with that.

So let’s make it easier to handle.

Tagging the response with friendly names

Remember up top when I set up the hookNames array with the friendly(ish) names of the remote webhooks? That’s what we make use of now.

An Array#reduce loop over the unnamed responses to tag each with its original name:

/** @repeated
const hookNames = hooks.map( hook => hook.name );
const fetchers = hooks.map( hook => FBHttp.fetch(hook.url) );

Promise.all( fetchers )
.then( responses => responses.map( resp => resp.json() ) )
.then( jsonparsers => Promise.all( jsonparsers ) )
*/
.then( results => results.reduce( (acc,result,idx) => {
   acc[hookNames[idx]] = result;
   return acc;
}, {})
)
.then( success )
.catch( failure );

Returning a much friendlier aggregate response to Marketo:

{
    "response": {
        "emailValidate": {
            "emailExists": true,
            "mxProvider": "Office 365"
        },
        "firmEnrich": {
            "sector": "Manufacturing",
            "size": "500-1000",
            "GSA": true
        },
        "reCAPTCHA": {
            "isHuman": true
        }
    }
}

Now, that response looks perfect for Response Mappings! And indeed it is. Too perfect.

Don’t forget error handling

The remaining problem is uptime of those 3 independent remote services can’t be guaranteed.

Say the Email Verification service suddenly became unavailable, like they changed their domain name and didn’t tell you. But the other 2 services were up and running.

Then you’d get a JSON response like so, with just the error:

{
    "errorMessage": "request to https://www.emailverifysvc.example?email=sandy%40teknkl.com failed, reason: getaddrinfo ENOTFOUND www.emailverifysvc.example",
    "errorType": "FetchError",
    "stackTrace": []
}

The reason for this is simple — and not specific to FlowBoost. It’s the way JavaScript’s standard Promise.all method works when processing a list of Promises:

  • if all the Promise-d requests complete successfully, then (literally, then) you get all the successful results
  • if one or more of the requests fails, an error is caught (catch-ed) with the error from the first failed request

That default behavior has caused many a dev to tear their hair out.

Of course, what we want to happen in the event that a service is down is only that service returns an error, and the others succeed as usual.

So the finishing touch is to make sure to catch individual errors. Here’s the final code:

const hooks = [
  { 
	"name" : "emailValidate",
	"url" : "https://www.emailverifysvc.example?email=" + encodeURIComponent({{Lead.Email Address}})
  },
  {
	"name" : "firmEnrich",
	"url" : "https://www.firmographicsvc.example?company=" + encodeURIComponent({{Company.Company Name}})
  },
  {
    "name" : "reCAPTCHA",
    "url" : "https://www.recaptchaverify.example?response=" + encodeURIComponent({{Lead.LastRecaptchaResponse}}) + "&secret=" + encodeURIComponent({{My.RecaptchaSecret}})
  }
];

/* --- NO NEED TO TOUCH BELOW THIS LINE! --- */

const hookNames = hooks.map( hook => hook.name );
const fetchers = hooks.map( hook => FBHttp.fetch(hook.url) );
      
Promise.all( fetchers.map( fetcher => fetcher.catch( JSON.stringify )) )
.then( responses => responses.map( resp => resp.json ? resp.json() : Promise.resolve(resp) ) )
.then( jsonparsers => Promise.all( jsonparsers.map( parser => parser.catch( JSON.stringify )) ) )
.then( results => results.reduce( (acc,result,idx) => {
   acc[hookNames[idx]] = result;
   return acc;
}, {})
)
.then( success )
.catch( failure );

Now, even if a webhook fails, we get the success results from the others, plus an embedded note about the error:

{
    "response": {
        "firmEnrich": {
            "sector": "Manufacturing",
            "size": "500-1000",
            "GSA": true
        },
        "reCAPTCHA": {
            "isHuman": true
        },
        "emailValidate": "{\"name\":\"FetchError\",\"message\":\"request to https://www.emailverifysvc.example?email=sandy%40teknkl.com failed, reason: getaddrinfo ENOTFOUND www.emailverifysvc.example\",\"type\":\"system\",\"errno\":\"ENOTFOUND\",\"code\":\"ENOTFOUND\"}"
    }
}

There are always new areas to optimize in Marketo!

Notes

[1] Or Python, C#, or the experts would say Erlang. But if you speak Erlang I doubt you need any pointers from this blog.

[2] The reasons being the different outbound permissions of FlowBoost Community & Standard vs. Pro.☺