Active Issue Alert – Blackbaud Payment Services, Blackbaud Merchant Services, Blackbaud Checkout - #2018021101
Incident Report for Blackbaud
Resolved
We have implemented a fix and have confirmed services have been restored to all affected customers. We are closely monitoring the affected service over the next 24 hours. Please continue to reference this page for the most recent status
Posted Feb 12, 2018 - 01:39 EST
Update
We have implemented a fix and have confirmed services have been restored to most affected customers. A small set of customers may still be experiencing issues. We are closely monitoring the affected service over the next 24 hours. Please continue to reference this page for the most recent status
Posted Feb 12, 2018 - 00:52 EST
Monitoring
We have implemented a fix and have confirmed services have been restored to all affected customers for over 1 hour. We are continuing to take preventative steps to ensure this does not reoccur. We are closely monitoring the affected service over the next 24 hours. Please continue to reference this page for the most recent status.
Posted Feb 11, 2018 - 23:40 EST
Update
Our engineers have implemented a solution that has resolved the payments issue for the majority of all payment services customers. However, there are still a few services that a small subset of some customers may not yet be able to access. We are working diligently to resolve this issue for you and expect a full restore of services for all customers soon. Customers that are continuing to receive errors and use use Internet Protocol (IP) address-based firewall rules may need to update whitelisting. Information on whitelisting can be found at https://maintenance.blackbaud.com/incidents/q6n995g1nkwk and https://kb.blackbaud.com/articles/Article/50865. Please continue to reference this page for the most recent status. Next update in 1 hour.
Posted Feb 11, 2018 - 22:01 EST
Update
Engineers are still working diligently to track down and apply fixes to the remaining affected services. We are very close to full service restoration at this time, and the affected customers are a very small subset. Most customers should be able to process payments and use all features and services of their hosted applications. We will continue to provide frequent updates, and will notify you as soon as we have confirmation of a full restore of services. Next update in 1 hour. Please continue to reference this page for the most recent status.
Posted Feb 11, 2018 - 21:13 EST
Update
Engineers are still working diligently to track down and apply fixes to the remaining affected services. We are very close to full service restoration at this time. We will continue to provide frequent updates, and will notify you as soon as we have confirmation of a full restore of services. Next update in 1 hour. Please continue to reference this page for the most recent status.
Posted Feb 11, 2018 - 20:12 EST
Update
Engineers successfully diagnosed the issue and are currently working to to apply the final fixes for the remaining affected services. We are very close to full service restoration at this time . Currently, we still expect it to take less than an hour for services to fully restore. We will continue to provide frequent updates, and will notify you as soon as we have confirmation of a full restore of services Please continue to reference this page for the most recent status.
Posted Feb 11, 2018 - 19:19 EST
Update
Our Engineers have successfully completed testing of a potential fix to the remaining affected services. We are now seeing payments process for some of the remaining affected services. Our Engineers are in the process of deploying this fix across the remaining affected environment and expect it to take up to another hour for services to fully restore. We will continue to post hourly updates to this page, with more frequent updates as information is available.
Posted Feb 11, 2018 - 18:00 EST
Update
Our Engineers have successfully completed testing of a potential fix to the remaining affected services. We are now seeing payments process for some of the remaining affected services. Our Engineers are in the process of deploying this fix across the remaining affected environment and expect it to take up to another hour for services to fully restore. We will continue to post hourly updates to this page, with more frequent updates as information is available.
Posted Feb 11, 2018 - 17:17 EST
Update
In a prior update, we confirmed that our engineers successfully diagnosed the issue and were working to fully restore the remaining affected services. We have identified the root cause and are making progress towards services restoration. Currently, we still expect it to take up to 1-2 hours for services to fully restore. As it gets closer to the ETA for service restoration, we will provide additional updates. We will continue to provide hourly updates, with more frequent updates as information is available. Please continue to reference this page for the most recent status.
Posted Feb 11, 2018 - 17:00 EST
Update
In a prior update, we confirmed that our engineers successfully diagnosed the issue and were working to fully restore the remaining affected services. We have identified the root cause and are making progress towards services restoration. Currently, we still expect it to take up to 1-2 hours for services to fully restore. As it gets closer to the ETA for service restoration, we will provide additional updates. We will continue to provide hourly updates, with more frequent updates as information is available. Please continue to reference this page for the most recent status.
Posted Feb 11, 2018 - 16:12 EST
Update
In our last update to the ongoing service impact to some of our customers using Blackbaud Payment Services, Merchant Services or Checkout, we confirmed that we have fully restored services to a portion of the affected services and were actively working on the fix to the remaining degraded services. We are currently implementing the fix to the remaining degraded services, and it is expected to take up to 2 hours to fully restore services. Our Engineers continue to to work to have your full service restored as quickly as possible. We will provide the next update in 1 hour. Please continue to reference this page for updates.
Posted Feb 11, 2018 - 16:00 EST
Update
In our last update to the ongoing service impact to some of our customers using Blackbaud Payment Services, Merchant Services or Checkout, we confirmed that we have fully restored services to a portion of the affected services and were actively working on the fix to the remaining degraded services. We are currently implementing the fix to the remaining degraded services, and it is expected to take up to 2 hours to fully restore services. Our Engineers continue to to work to have your full service restored as quickly as possible. We will provide the next update in 1 hour. Please continue to reference this page for updates.
Posted Feb 11, 2018 - 15:01 EST
Identified
We are aware that some of our Payment Services, Merchant Services or Checkout customers are experiencing an issue when processing transactions. Our engineers have successfully diagnosed the issue and we have the appropriate engineers available to address it for you. We have successfully restored services to a portion of the affected services, and are actively working on the fix to the remaining degraded services. We will continue working to have your full service restored as quickly as possible. Please continue to reference this page for updates.
Posted Feb 11, 2018 - 14:14 EST
Update
We have identified an issue impacting some of our customers. For all products that use Blackbaud Payment Services, Blackbaud Merchant Services or Blackbaud Checkout, intermittent errors are presenting when attempting to receive or process donations. Our Engineers have isolated the potential reason for outage and are working to fully restore services as quickly as possible. Please continue to reference this page for updates.
Posted Feb 11, 2018 - 13:47 EST
Investigating
We have identified an issue impacting some of our customers causing intermittent slowness, errors or other service issues. Our engineers are still working to diagnose the cause and we have the appropriate engineers working to get your full service restored as quickly as possible. Please continue to reference this page for updates.
Posted Feb 11, 2018 - 12:30 EST
Identified
We have identified an issue impacting some of our customers causing intermittent slowness, errors or other service issues. Our engineers are still working to diagnose the cause and we have the appropriate engineers working to get your full service restored as quickly as possible. Please continue to reference this page for updates.
Posted Feb 11, 2018 - 11:59 EST
Investigating
We have identified an issue impacting some of our customers causing intermittent slowness, errors or other service issues. Our engineers are still working to diagnose the cause so that they can resolve it as quickly as possible for you. Please continue to reference this page for updates.
Posted Feb 11, 2018 - 11:28 EST
This incident affected: Email Services.