Removing duplicate items from a multivalued string field using FlowBoost

I’ve got a draft post (always saying that but it’s always true!) titled Working with multivalued string fields in FlowBoost covering a huge list of use cases.

In the meantime, someone asked a direct question so let’s do that one right now:

We append to a multivalued field using Change Data Value. Unfortunately, this means we often end up with repeated values like apples;oranges;peaches;oranges which is sloppy and hard to read. Can I reduce that to only the unique values?

If you used FlowBoost to do a “smart append” instead of using Change Data Value, you wouldn’t have this situation. But let’s assume existing values look like this and you want to fix ’em up.

Lots of different ways to remove dupes in a multivalued string using JS. Here’s a straightforward-ish one:

let delimiter = ";";

let ws = "\\\\s*";
let splitter = new RegExp(ws + delimiter + ws);
let original = {{Lead.Product Interest}}.split(splitter);

let deduped = Array.from(new Set(original));

dedupedStringified = deduped.join(";");

Then you map dedupedStringified back to the field in your Webhook » Response Mappings.[1] Presto!

How it works

First, split() the String field on the delimiter, creating an Array. You can set delimiter to a comma or pipe or whatever, though Marketo uses ; for picklists and multi-selects so it’s best to standardize. Any whitespace (ws) around the delimiter is also removed for cosmetic purposes.

Next, create a Set with new Set(Array) which removes dupes automatically. Unlike Arrays, Sets always have unique items.

Next, turn the Set back into a regular Array with Array.from(Set).

Finally, join() back into a String to put into the single Marketo field.

With huge lists (I mean like 100,000+ items) this wouldn’t be efficient and would make CS pedants groan. In a Marketo context it’s impossible to have that many values, so don’t worry about it.

Oh, one more thing

You might wonder why I didn’t write

let ws = "\\s*";

but quadruple-escaped \\\\s* instead.

Marketo’s webhook payload box is a little too actively involved. It pre-parses the whole payload once (though IMO it shouldn’t touch the static parts of the payload at all).

Thus it’ll turn \\ into \ before sending to FlowBoost (or any remote service). So you need to put \\\\ in the box so the service receives \\.

It’s frustrating because if you’re using backslashes anywhere in your code, you can’t copy code from Postman or your browser directly to Marketo. You have to double any slashes along the way.

Notes

[1] Note there’s no let before dedupedStringified = ... since we want that to be a global variable.

As you hopefully recall from other posts, only global variables are returned in the FlowBoost response. It would also be okay to use var dedupedStringified = .... But neither let nor const creates a global, meaning such variables are for use internally within FlowBoost but won’t be seen by Marketo.