We have deployed a new DB fix and are currently monitoring

unfortunately yesterdays issues are reappearing now intermittently. We are investigating this to get to the bottom of it

This incident has been resolved

our internal tests confirmed everything is working now, we will monitor for some time before closing this incident

we have rebooted our DB and caching servers, things seem to stabilize

We are currently experiencing an issue where for some of our customers the direct upload of an episode to YouTube fails. We are working on a fix for this. In the meantime, the workaround is to manually download the video file and upload directly to your YouTube channel.

This does not affect YouTube simulcasting

the direct impact on us from these third party issues have been resolved

[01:58 PM PST] We can confirm significant recovery for the DNS resolution errors for Cloudfront distributions.

[01:46 PM PST] We are seeing early signs of recovery, and continue to work toward full recovery.

From our provider: [01:40 PM PST] We can confirm errors for DNS resolution for some Cloudfront distributions. During this time, customers may receive an NXDOMAIN response. Additionally, customers may also experience delayed propagation for changes to CloudFront distributions. We have identified the root cause of the issue and are actively working on multiple paths to resolving the errors. We have verified that our initial mitigation effort on a portion of the affected subsystem was successful, and we are actively working toward performing that mitigation across the fleet. We recommend customers continue to retry any failed requests while we work toward mitigation. A few services that use CloudFront distributions for delivering content may also be affected at this time.

Our service provider is currently experiencing an incident that might cause  some of our web services to fail loading

Details from our provider (AWS): [01:15 PM PST] We are investigating DNS resolution failures for some specific Cloudfront distributions. We are actively investigating and will provide additional information in the next 30-60 minutes.

this incident has been resolved

the new API was deployed and tested, we are monitoring until Oct 30.

We’re currently upgrading our Facebook API integration for simulcasting. During this time, new Facebook connections cannot be established. Existing connections are unaffected. We expect this to be resolved by Wednesday, October 29.

As a temporary workaround, follow this article to manually add and rtmp connection (see note about custom rtmp at the bottom).

Thank you for your patience.

we have rolled back a recent change that has caused the issue, everything is back online

Media centers, such as ebs.sermon.net, are currently unavailable, we are investigating. Embedded players, live streaming, simulcasting are not affected

We have monitored over the weekend and can confirm it was resolved

We have restored all services and are currently monitoring

Simulcasting recovered, currently watching streams through mediacenter is impacted

we are restarting our services after getting rid of the DB spike

We are experiencing a major DB spike which causes sermon studio as well as media centers and live streaming to be impacted.

We are investigating