Active Issue Alert – Online Express - 000306021
Incident Report for Blackbaud
Resolved
An issue impacting some of our customers is now resolved and services have been restored. If you have any questions about status in the future, you can always get updates at this page. Thank you.
Posted Nov 26, 2021 - 19:56 EST
Update
Engineers have determined it is not possible for Blackbaud to send donation acknowledgement emails, for donations made between 2 p.m. on Tuesday, November 16 to 12 a.m. ET on Thursday November 18, to constituents on your behalf. To manually send those emails, please follow the instructions in this Knowledgebase article: https://kb.blackbaud.com/knowledgebase/articles/Article/197918.
Posted Nov 22, 2021 - 16:19 EST
Monitoring
As of 10 p.m. ET on Sunday, November 21, 2021, the issue with duplicate transactions being created when manually sending acknowledge emails has been resolved, and you may now resume resending acknowledgement emails through the Online Express plugin. If you have duplicate transactions that need to be cleaned up, please follow the instructions in this Knowledgebase article, https://kb.blackbaud.com/knowledgebase/articles/Article/197918.



While we continue to explore if we can bulk resend acknowledgement emails on your behalf for donations made between 2 p.m. ET on Tuesday, November 16, 2021 through 12 a.m. ET on Thursday, November 18, 2021 you can manually resend the emails by following the instructions in this Knowledgebase article, https://kb.blackbaud.com/knowledgebase/articles/Article/197918. If you choose to send emails on your own, we will exclude those donors if we are later able to send emails in bulk on your behalf. We will update this page once we have more information. Again, we apologize and thank you for your patience.
Posted Nov 22, 2021 - 11:10 EST
Update
As of 4:50 PM ET, the privacy policy and unsubscribe links included in acknowledgement emails are now working and accessible. We are still investigating the issue with duplicate transactions being created within the plugin when manually sending acknowledge emails. Please continue to refrain from sending acknowledgement emails through the Online Express plugin until this is resolved.

We apologize for this inconvenience and sincerely thank you for your patience. We will update this page once we have more information on our investigation into the duplicate transactions.
Posted Nov 19, 2021 - 17:42 EST
Update
In the previous status update we shared an alternative way for you to resend the acknowledgement emails for donations made between 2 p.m. ET on Tuesday through 12 a.m. ET on Thursday that were not sent. However, we have identified that duplicate transactions are created within the plugin when using this alternative. Until further notice, we ask all users refrain from resending acknowledgement emails through the Online Express plugin. If you did resend those missing acknowledgement emails and now have duplicate transactions, please follow the instructions in this KB article, https://kb.blackbaud.com/knowledgebase/articles/Article/197918 . We continue to look into the possibility of resending those emails, in bulk, on your behalf.

As noted in the previous status update, your donors should receive an acknowledgement email after making a donation. However, because of the fix we implemented, constituents are unable to access the privacy policy and unsubscribe links that are included in the acknowledgement email they receive. Engineers continue to investigate this and are working to fix that issue.

We will update this page once we have more information on our investigation.
Posted Nov 19, 2021 - 12:55 EST
Update
Going forward, your donors should receive an acknowledgement email after making a donation. However, because of the fix we implemented, constituents are unable to access the privacy policy link that is included in the acknowledgement email they receive. Engineers are aware of this and are working to fix that issue. Acknowledgement emails for donations made between 2 p.m. ET on Tuesday through 12 a.m. ET on Thursday were not sent. If you want to send those acknowledgement emails, please follow the instructions in this Knowledgebase article, https://kb.blackbaud.com/knowledgebase/articles/Article/197918. We are also looking into the possibility of sending those emails, in bulk, on your behalf. If you choose to send emails on your own, your donors should not receive duplicate emails if we are able to send those in bulk. We will update this page once we have more information on our investigation.
Posted Nov 18, 2021 - 16:06 EST
Update
As of 12:00 AM EST, donation acknowledgment emails should be sent for donations moving forward. We are working on a plan to best support customers who need to send acknowledgment emails for donations made during the time of the incident. We will update this status page with that information once we have more details to share.
Posted Nov 18, 2021 - 01:23 EST
Update
As of 6:00 p.m. ET, the processes that are expected to fix the issue are running as expected but will still take several hours to complete. Once those processes finish running, the issue should be resolved. In terms of sending donation acknowledgment emails, you have no action to take right now. Our team is fully focused on resolving this issue for you. Once the issue is resolved, we will focus our efforts on how best to send acknowledgment emails for donations made while this issue was occurring. Please subscribe to this alert so you can receive real-time updates once additional information is shared.
Posted Nov 17, 2021 - 18:11 EST
Update
As of 12:45 PM ET, progress for resolving this issue continues as expected. The processes that are expected to fix the issue will still take several hours to complete, but the team has not run into any issues so far. In terms of donation acknowledgement emails, you have no action to take right now. Those emails are not being sent to your donors, but our team is looking into the possibility of sending those emails, in bulk, once this issue is resolved. Please subscribe to this alert so you can receive real-time updates once additional information is shared.
Posted Nov 17, 2021 - 12:47 EST
Update
As we’ve investigated, we have determined that this issue is preventing email statistics from updating within Online Express. Additionally, while donations continue to process successfully, donation acknowledgement emails are not being sent. Acknowledgement emails, for donations made since this issue started at 12 p.m. ET on Tuesday, will not be sent to donors. Engineers are currently running several processes to resolve this issue; however, we don’t expect this issue to be resolved until Wednesday evening at the earliest. The team is looking into the possibility of sending donation confirmation emails retroactively. We will update you as we investigate that possibility. Please subscribe to this alert so you can receive real-time updates once additional information is shared.
Posted Nov 17, 2021 - 10:31 EST
Identified
We have identified an issue impacting some of our customers. Our engineers are still working to diagnose the cause and we have all the appropriate engineers available to get your full service restored as quickly as possible. Please continue to reference this page for updates. The next update will occur at 12pm EST on 11/17/2021.
Posted Nov 17, 2021 - 08:18 EST
Investigating
We have identified an issue impacting some of our customers. Our engineers are still working to diagnose the cause and we have all the appropriate engineers available to get your full service restored as quickly as possible. Please continue to reference this page for updates. The next update will occur at 8am EST on 11/17/2021.
Posted Nov 16, 2021 - 23:01 EST
Identified
We have identified an issue impacting some of our customers causing intermittent slowness, errors or other service issues. Our engineers have successfully diagnosed the issue and have all the appropriate engineers working to address it for you. We hope to have your full service restored as quickly as possible. Please continue to reference this page for updates.
Posted Nov 16, 2021 - 22:16 EST
Update
We have identified an issue impacting some of our customers. Our engineers are still working to diagnose the cause so that they can resolve it as quickly as possible for you. Please continue to reference this page for updates.
Posted Nov 16, 2021 - 21:44 EST
Investigating
We have identified an issue impacting some of our customers causing errors when attempting to log in or use features within the affected service. Our engineers are diagnosing the event and working to resolve it for you as quickly as possible. Please continue to reference this page for updates.
Posted Nov 16, 2021 - 20:58 EST
This incident affected: Blackbaud Online Express™ (Blackbaud Online Express™).