tag:status.blitzz.co,2005:/historyBlitzz Status - Incident History2024-03-28T22:22:48-07:00Blitzztag:status.blitzz.co,2005:Incident/182605432023-08-29T10:22:25-07:002023-08-29T10:22:25-07:00Reports and Dashboards are not refreshed<p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>10:22</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>10:02</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>11:00</var> PDT</small><br><strong>Identified</strong> - We have identified an issue refreshing the Usage Dashboards for some of our customers. Our team is working on this issue and will provide an update as soon as this is fixed. As a result of this, the data will only be visible until Aug 23 2023 10:00AM UTC.</p>tag:status.blitzz.co,2005:Incident/174085372023-05-29T20:51:51-07:002023-05-29T20:51:51-07:00Error Loading Blitzz App<p><small>May <var data-var='date'>29</var>, <var data-var='time'>20:51</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>05:01</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>05:01</var> PDT</small><br><strong>Monitoring</strong> - We have identified the source of the issue. While our team continues to work on a full resolution, we are seeing a stable performance in the app and will provide an update as soon as more information becomes available.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>04:52</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>04:50</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>04:15</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>04:15</var> PDT</small><br><strong>Investigating</strong> - Some users may face a longer time loading the web app or an error loading the app. Our engineering team is notified and is working on increasing the capacity.</p>tag:status.blitzz.co,2005:Incident/94371162022-03-01T18:44:36-08:002022-03-01T18:44:36-08:00Unable to start video session<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>18:44</var> PST</small><br><strong>Resolved</strong> - We are no longer experiencing failures for joining a room with Video. This incident has been resolved.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>10:17</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>09:56</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>09:50</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>09:41</var> PST</small><br><strong>Update</strong> - We are investigating failures when customers are attempting to create or join session. We expect to provide another update in 30 minutes or as soon as more information becomes available.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>09:22</var> PST</small><br><strong>Update</strong> - We are experiencing some connectivity issues. Affected customers experienced failed calls and messages. Our engineers are working to restore all the services shortly.
<br />We apologize for the inconvenience.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>09:21</var> PST</small><br><strong>Investigating</strong> - We have identified an issue with starting a new video session and starting a call. Our team is investigating and will provide an update soon.</p>tag:status.blitzz.co,2005:Incident/87977772021-12-08T23:21:39-08:002022-03-01T09:20:30-08:00Unable to start video session<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>23:21</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>23:07</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>19:48</var> PST</small><br><strong>Investigating</strong> - We have identified an issue with starting a new video session and starting a call. Our team is investigating and will provide an update soon.</p>tag:status.blitzz.co,2005:Incident/87843062021-12-07T19:32:33-08:002021-12-07T19:32:33-08:00Reports not working<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>19:32</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>18:49</var> PST</small><br><strong>Update</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>10:17</var> PST</small><br><strong>Investigating</strong> - We have identified an issue with Reports and it is not loading currently. Our team is investigating and will provide an update soon.</p>tag:status.blitzz.co,2005:Incident/81606162021-10-05T14:41:59-07:002021-10-05T14:41:59-07:00Unable to start a video session<p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>14:41</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>13:39</var> PDT</small><br><strong>Monitoring</strong> - We experienced a brief period of connectivity issues impacting some of our customers. We are currently monitoring the situation.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>13:32</var> PDT</small><br><strong>Investigating</strong> - We have identified an issue with starting a new video session and starting a call. Our team is investigating and will provide an update soon.</p>tag:status.blitzz.co,2005:Incident/67880132021-04-19T20:54:20-07:002021-04-19T20:54:20-07:00SMS Delivery Delays to Canada Rogers/Fido Network<p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>20:54</var> PDT</small><br><strong>Resolved</strong> - Wireless outage has fully restored and the SMS backlog was deemed cleared at 22:30 EDT.<br /><br />We sincerely apologize for any inconvenience caused.</p><p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>20:47</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>10:29</var> PDT</small><br><strong>Identified</strong> - Our monitoring systems have detected an intermittent issue with a delay in SMS delivery to Canada Rogers Wireless/Fido Network. Thank you for your patience while we are working with our partners and the operator to get this resolved. We are following it closely.<br /><br />More Details: <br />https://status.sinch.com/incidents/xlx5sr63ntwp<br />https://status.twilio.com/incidents/jpt99j323h41<br /><br />Please contact support@blitzz.co if you require further information or assistance.</p>tag:status.blitzz.co,2005:Incident/66594162021-04-01T17:45:52-07:002021-04-02T06:53:52-07:00Error Loading Blitzz App<p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>17:45</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>16:30</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.<br /><br />Summary of Impact: Between approximately 21:30 UTC and 22:30 UTC on 01 Apr 2021, our service provider, MS Azure experienced intermittent issues due to a DNS outage.<br /><br />Preliminary Root Cause: Microsoft noticed a spike in DNS traffic. They are continuing to investigate the underlying cause.<br /><br />Mitigation: Microsoft engaged resilient DNS capabilities to absorb the spike in DNS traffic.<br /><br />More details: https://status.azure.com/en-us/status/history/</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>15:18</var> PDT</small><br><strong>Identified</strong> - We have identified a DNS issue with our hosting provider, Microsoft Azure. We are working with them to fix this issue.<br /><br />More info: <br />https://status2.azure.com/</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>15:03</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>15:02</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>14:58</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>14:49</var> PDT</small><br><strong>Investigating</strong> - Some users may face a longer time loading the web app or an error loading. Our engineering team is notified and is working on increasing the capacity.</p>tag:status.blitzz.co,2005:Incident/66021002021-03-25T09:39:04-07:002021-03-25T09:39:04-07:00Data Refresh Issue for Dashboards<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>09:39</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>04:25</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>10:54</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>10:15</var> PDT</small><br><strong>Investigating</strong> - We are experiencing a data refresh issue for the dashboards. Some reports may not reflect the latest call data. Our engineers have been notified of this issue and are working to identify the root cause.<br /><br />We apologize for the inconvenience. If you have any questions or concerns please do not hesitate to email support@blitzz.co.</p>tag:status.blitzz.co,2005:Incident/64066102021-02-26T08:02:16-08:002021-02-26T08:02:16-08:00Error sending SMS messages<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>08:02</var> PST</small><br><strong>Resolved</strong> - We are no longer experiencing SMS Delivery issues.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>06:40</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>06:38</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>05:30</var> PST</small><br><strong>Investigating</strong> - Our monitoring systems have detected a potential issue with SMS Delivery through our partner, Twilio. Our engineering team has been alerted and is actively investigating. We will update as soon as we have more information.<br /><br />More details on the incident: https://status.twilio.com/incidents/zmz5c9prz8mm</p>tag:status.blitzz.co,2005:Incident/58277042020-12-18T00:01:53-08:002020-12-18T00:01:53-08:00SMS Carrier Partner Maintenance - USA and CA Short Code<p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>00:01</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>21:01</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>12:09</var> PST</small><br><strong>Scheduled</strong> - Upcoming scheduled maintenance notice<br /><br />Our SMS carrier partner for Short Code in USA and Canada is conducting a scheduled maintenance from 17 December 2020 at 21:00 PST until 17 December at 2020 23:59 PST. <br /><br />During the maintenance window, SMS messages from Blitzz to USA/Canada will be routed using long code numbers to avoid any delays and failures. Our support team will be actively monitoring SMS delivery over long codes. If there are any delays or failures, we will contact you directly.<br /><br />No action is required from your side and we do not expect any down time during this maintenance window.<br /><br />On behalf of our North American SMS carrier partner, we appreciate your patience and apologize for any inconvenience during this maintenance period. <br /><br />If for any reason the maintenance is extended or cancelled we will advise you accordingly.<br />Please contact support@blitzz.co should you require further information or assistance.</p>tag:status.blitzz.co,2005:Incident/57704312020-12-12T14:09:06-08:002020-12-12T14:09:06-08:00SMS Delivery Failures to Canadian Short Codes<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>14:09</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>08:50</var> PST</small><br><strong>Update</strong> - We have temporarily switched to using a group of long code instead of a shortcode for SMS delivery. Our support team is continuously monitoring delivery over long codes and there are no issues.<br /><br />We are also observing improvements in SMS delivery when sending to and from Canadian short code number. We will continue to monitor to ensure full-service recovery before switching back to shortcode. We expect to provide another update in 2 hours or as soon as more information becomes available.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>06:42</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are now monitoring to ensure full-service recovery. We expect to provide another update in 2 hours or as soon as more information becomes available.</p>tag:status.blitzz.co,2005:Incident/56701972020-11-27T06:54:34-08:002020-11-27T07:39:15-08:00SMS Delivery Delays to Canada Over Short Code<p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>06:54</var> PST</small><br><strong>Resolved</strong> - We are no longer experiencing delivery failures when sending messages to Canada over shortcodes. This incident has been resolved.</p><p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>06:54</var> PST</small><br><strong>Investigating</strong> - We are experiencing SMS delivery delays when sending messages to Canada over shortcodes. Our engineers are working with our carrier partner to resolve the issue. We will provide another update in 1 hour or as soon as more information becomes available.</p>tag:status.blitzz.co,2005:Incident/55295672020-11-11T01:00:34-08:002020-11-11T01:00:34-08:00CANADA SMS Carrier Partner Maintenance<p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>01:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>22:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>22:13</var> PST</small><br><strong>Scheduled</strong> - Our SMS carrier partner in the CANADA is conducting a planned maintenance from 10 November 2020 at 22:00 PST until 11 November 2020 at 01:00 PST. During the maintenance window, there could be intermittent delays delivering SMS to and from CANADA handsets via CANADA short code.<br /><br />MAINTENANCE PERIOD<br /><br />US Pacific: Nov 10, 22:00 - Nov 11, 01:00<br />US Central: Nov 11, 00:00 - Nov 11, 03:00<br />US Eastern: Nov 11, 01:00 - Nov 11, 04:00<br />UTC: Nov 11, 06:00 - Nov 11, 09:00</p>tag:status.blitzz.co,2005:Incident/49957832020-09-02T01:49:12-07:002020-09-02T06:49:45-07:00Audio Loss during Call Recording on Chrome 85 or Above<p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>01:49</var> PDT</small><br><strong>Resolved</strong> - The fix has been rolled out to the production platform, and we are no longer observing audio loss with Video call recording where customers are using Chrome 85 or above. This issue has been resolved and all services are operational now.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>22:29</var> PDT</small><br><strong>Identified</strong> - Our engineers have implemented and verified the fix. Now running regression tests in the staging environment and preparing for a rollout.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:24</var> PDT</small><br><strong>Investigating</strong> - Note: this does not impact group calls that are not being recorded.<br /><br />Our engineers are investigating a potential issue with Call Recordings where customers using Chrome 85 or above may experience intermittent audio loss. The fix is identified and is under testing. We expect to provide another update in 1 hour or as soon as more information becomes available.</p>tag:status.blitzz.co,2005:Incident/49266262020-08-24T14:37:59-07:002020-08-24T14:37:59-07:00We are experiencing intermittent SMS delivery delays when sending messages to these networks:<p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>14:37</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>16:00</var> PDT</small><br><strong>Identified</strong> - Our engineers are working with our carrier partner to resolve the issue. We will provide an update as soon as we have additional information.<br /><br />Please click here for detailed updates:<br />https://status.twilio.com/incidents/0yj8c6lmnzv0</p>tag:status.blitzz.co,2005:Incident/47266602020-07-28T15:29:19-07:002020-07-28T15:29:19-07:00Technical Incident - Email Invite delays<p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>15:29</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>13:21</var> PDT</small><br><strong>Update</strong> - The issue resulting in email delivery delays has been resolved. The systems are now operating normally. We will continue to monitor the situation before we mark the incident as fully resolved.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>12:06</var> PDT</small><br><strong>Monitoring</strong> - We have identified the source of the issue that resulted in email delivery delays. While our team continues to work on a full resolution, we are seeing a stable performance in email delivery in the past 1 hour and will provide an update as soon as more information becomes available.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>10:38</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>10:35</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>10:07</var> PDT</small><br><strong>Investigating</strong> - Some users may be experiencing issues regarding email invites. We are currently investigating the issue and will provide updates as they become available. We sincerely apologize for any inconvenience this may cause. Please reach out to our Support Team at https://help.blitzz.co with any questions.</p>tag:status.blitzz.co,2005:Incident/40855022020-05-11T07:35:37-07:002020-05-11T07:35:37-07:00Reporting is down<p><small>May <var data-var='date'>11</var>, <var data-var='time'>07:35</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>07:34</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>07:32</var> PDT</small><br><strong>Update</strong> - We have resolved the issue and reporting is operational now.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>07:29</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:status.blitzz.co,2005:Incident/40048582020-04-24T06:30:00-07:002020-04-30T21:40:54-07:00Reports are not accessible<p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>06:30</var> PDT</small><br><strong>Resolved</strong> - Between approximately 2020-04-24 1336 UTC and 2020-04-24 1515 UTC, Blitzz Reporting services experienced unplanned downtime due to an AWS outage, which is highly unusual. AWS Quicksight team has confirmed the issue is resolved at their end and all services should be running normal up again.<br /><br />No action is required from your end.</p>tag:status.blitzz.co,2005:Incident/26721942019-07-08T21:09:37-07:002020-04-30T21:40:54-07:00Major Azure Outage Due to DNS Migration Issue<p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>21:09</var> PDT</small><br><strong>Resolved</strong> - Our data processing infrastructure is running behind which is causing inaccuracies in the reporting tools. No data has been lost and the system should be caught up shortly.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>21:04</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>05:29</var> PDT</small><br><strong>Investigating</strong> - A DNS outage affected many services in Microsoft Azure between 19:43 and 22:35 UTC on May 2, 2019. The outage affected many services within Microsoft Azure; such as Compute, Storage, App Service, Azure Active Directory, and Azure SQL Database services. Additional Microsoft services were affected as well, including Microsoft 365, Dynamics, Azure DevOps, and more. As you can tell this was a large outage!</p>tag:status.blitzz.co,2005:Incident/26720172019-07-08T05:12:16-07:002020-04-30T21:40:54-07:00SMS is not sent<p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>05:12</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>05:11</var> PDT</small><br><strong>Monitoring</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>05:09</var> PDT</small><br><strong>Identified</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>05:03</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>04:46</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>04:44</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>04:43</var> PDT</small><br><strong>Update</strong> - Based on further monitoring, we have observed similar delays to US Virgin Mobile handsets as a part of this incident.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>04:43</var> PDT</small><br><strong>Investigating</strong> - We are experiencing delays in delivering SMS messages from a subset of US shortcodes to US Sprint handsets. Our engineers are working with our carrier partner to resolve the issue. We will provide an update as soon as we have additional information.</p>