Updates about the GivingTuesday 2024 service disruptions

December 4, 2024

‍

I wanted to personally apologize for the service disruptions you may have experienced on GivingTuesday while using our platform. We understand the significant investment of money, resources and time you dedicated to preparing for this pivotal day. You counted on a successful and seamless fundraising experience to launch the giving season, and for many of you, we fell short of the trust you placed in us on this critical occasion.

‍

Give Lively has powered nine years of GivingTuesdays. This was our first with a service disruption. We want to assure you we are diligently working to ensure this does not happen again. We want to share the following update:

‍

We want to alleviate some of your concerns

• We are certain the service disruptions were not related to a security breach or data loss. Your data is secure and protected.
• We can confirm that millions of dollars in donations made by tens of thousands of donors yesterday were unaffected by these disruptions.
• We are not currently experiencing issues with our service today. We are fully operational, with normal response times.
• We don't expect these disruptions to occur again.

What happened yesterday (12/3)

• We saw three periods of service disruption over GivingTuesday. We define a service disruption as slow load times, with a portion of visitors intermittently unable to reach our donation pages. 

What we did when we learned of the first disruption

• We immediately initiated our existing Crisis Response Protocol, which included live troubleshooting with members of the entire leadership team and our entire engineering team. This was an all-day, all-night effort, and continues today, right now.
• We immediately sent out a status update at status.givelively.org and on our support chat widget.
• We began a root cause analysis and worked on fixes.

What we've uncovered so far

• These issues revealed unforeseen infrastructure problems not found in "load tests" that we routinely run in the months and days leading up to GivingTuesday. Load tests are ways we push our system to mimic traffic we may see on days like GivingTuesday.
• Specifically, we experienced critical failures in how we serve pages to the public at scale. A bottleneck in our infrastructure made it difficult to keep up with demand.
• We released 3-4 attempts at fixes through this period. Although they did relieve pressure on our infrastructure, they did not solve the underlying problem.

What we are doing now

• The whole Give Lively team is continuing to investigate this issue.
• We are working on a root cause analysis.
• We are working on a remediation plan.

A note on our communications during this event

• We pushed updates to our status page and to our customer support chat widget. We know this was insufficient for many of you. We did not proactively send an email to our entire nonprofit population. I want you to know we are re-examining our communication protocols.

Give Lively is a small team of people entirely dedicated to helping the nonprofit community. It's our reason for existing. You trust us to run your critically important campaigns during this busy season. As we enter the last month of the year, and the end of the tax season, we want you to know that we are actively working to ensure our platform performs for you, as it has for many years now.

‍

Although this message does not have all the answers, I am committed to keeping you updated as we work through this difficult time. I will communicate with you again tomorrow, and as we have more to share.

‍

Sincerely,

‍

David DeParolesa

Chief Executive Officer / Give Lively

December 5, 2024

‍

Yesterday, I promised to share an update with you as we continue our investigation into the disruptions we experienced on GivingTuesday.

• We continue to be fully operational, with normal response times and no service interruptions.
• We believe the disruption on Tuesday was related to an unforeseen infrastructure bottleneck, specifically in how we perform "caching." Caching is meant to make pages load faster by optimizing the storage of frequently used information for quick access. We believe a critical system did the opposite — it was storing data inefficiently and caused a traffic jam that slowed our pages in ways our own testing did not catch. We are still investigating.
• The Give Lively engineering team is actively conducting a front-to-back audit of our infrastructure and code to ensure this does not happen again. This effort is all-encompassing and will be their sole priority until we feel confident we've reached a resolution.
• We have begun to identify outside services that can help in this process.

I hope transparency in our process can offer a first step to rebuilding the trust you placed in us during this important fundraising period.

As we have more information, I'll share it with you.

‍

Sincerely,

‍

David DeParolesa

Chief Executive Officer / Give Lively