Today's epically embarrassing (until we forget about it next week!) Marketo outage was unfortunately predictable — if you were tuned in to a certain underrated, overlong DNS outage in June.
Though the CEO's post-mortem mentions…
we renew thousands of domain name properties we own every year with precision
… in fact this Community thread shows some pretty glaring imprecision.
On that thread, several of us discussed the fact that the hostname social.mrk.to
(and in fact the whole domain mrk.to
) had disappeared. That domain is a production domain, and the downtime caused a total failure of social sharing functionality. (That wasn't obvious when merely viewing an affected LP, of course, so it wasn't widely known.)
Though tagged as P1, that issue took a full a week to fix, which (to me) suggests that no one was minding the DNS store.
This isn't to say the vast majority of Marketo's zones aren't properly managed (I don't really know either way, and we all hope there are zero problems like this in the future).
But to frame this DNS-related fail as the only one in recent memory is incorrect.
Just, as they say, sayin’. :)