tag:status.cludo.com,2005:/historyCludo Status - Incident History2024-03-29T06:23:58+01:00Cludotag:status.cludo.com,2005:Incident/203264592024-03-21T22:30:00+01:002024-03-22T11:52:29+01:00Search Outage for US Customers<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>22:30</var> CET</small><br><strong>Resolved</strong> - The issue was identified and resolved. The search in the US region failed due to an error in the underlying infrastructure. The outage lasted for approx. 2 hours.</p>tag:status.cludo.com,2005:Incident/200952832024-02-28T12:18:33+01:002024-02-28T12:18:33+01:00Search outage on EU<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>12:18</var> CET</small><br><strong>Resolved</strong> - Today's search outage was caused by a number of customers our customers in the EU experiencing a DDoS attack. Measures will be put into place to avoid similar scenarios in the future.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>11:20</var> CET</small><br><strong>Identified</strong> - We believe we have identified the issue and are currently working on a solution.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>09:33</var> CET</small><br><strong>Investigating</strong> - We are currently experiencing an outage on our EU search cluster. We are currently investigating this issue and working hard to get things back up and running as soon as possible. We apologize for any inconvenience that this must be causing.</p>tag:status.cludo.com,2005:Incident/200880042024-02-27T16:48:45+01:002024-02-27T16:48:45+01:00Analytics Unavailable for All Customers<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>16:48</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>14:51</var> CET</small><br><strong>Investigating</strong> - We are currently experiencing an issue where Analytics functionality is inaccessible for all customers. Our team is actively investigating the root cause of this disruption and working to restore full analytics functionality as soon as possible.</p>tag:status.cludo.com,2005:Incident/199395192024-02-08T19:03:50+01:002024-02-08T19:03:50+01:00Analytics Unavailable for Customers on US Cluster<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>19:03</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>15:10</var> CET</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>15:07</var> CET</small><br><strong>Investigating</strong> - We are currently experiencing an issue where Analytics functionality is inaccessible for customers on the US cluster. Our team is actively investigating the root cause of this disruption and working to restore full analytics functionality as soon as possible.</p>tag:status.cludo.com,2005:Incident/198386722024-01-27T11:06:53+01:002024-01-27T11:06:53+01:00Degraded performance for US crawlers<p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>11:06</var> CET</small><br><strong>Resolved</strong> - Crawlers are now running at a normal pace.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>16:42</var> CET</small><br><strong>Identified</strong> - As a result of the recent maintenance of our US cluster, crawlers waiting to start have queued up. This is expected behavior, and we expect this queue to be brought down in the next day.<br /><br />To clarify, crawlers are running - we're simply facing a delay when it comes to starting a new crawl.</p>tag:status.cludo.com,2005:Incident/198297482024-01-26T11:51:22+01:002024-01-26T11:51:22+01:00Planned maintenance for US search cluster<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>11:51</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>15:07</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>15:06</var> CET</small><br><strong>Scheduled</strong> - We are currently performing structural changes to our search cluster in the US. This is part of our ongoing efforts to completely eradicate any possibility of search being unavailable for customers located on the US cluster. This may result in partial availability of search during the next few hours, and we are monitoring the process closely to make sure we avoid any issues. This status page will be updated to show any further changes.<br /><br />We are aware of the inconvenience that this must be causing for you and your users, and getting our US search to a stable stage is our very highest priority. We thank you for your patience in this matter.<br /><br />-- <br />David Babayan, CTO</p>tag:status.cludo.com,2005:Incident/198183372024-01-25T10:17:12+01:002024-01-25T10:17:12+01:00Search outage on US cluster<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>10:17</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>01:08</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>23:39</var> CET</small><br><strong>Update</strong> - We are actively apply the fix now.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>17:57</var> CET</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>15:50</var> CET</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>15:30</var> CET</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cludo.com,2005:Incident/197979602024-01-23T03:16:04+01:002024-01-23T03:16:05+01:00Search Outage on US Cluster<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>03:16</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>21:19</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>21:07</var> CET</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>21:07</var> CET</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:status.cludo.com,2005:Incident/197953052024-01-22T16:56:16+01:002024-01-22T16:56:16+01:00Search outage on US cluster<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>16:56</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>16:36</var> CET</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>16:23</var> CET</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:status.cludo.com,2005:Incident/196887092024-01-12T16:03:12+01:002024-01-12T16:03:12+01:00Search API down (EU)<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>16:03</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>15:48</var> CET</small><br><strong>Monitoring</strong> - The systems are back up and we are monitoring stability.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>15:36</var> CET</small><br><strong>Investigating</strong> - It appears we are not quite stable yet and are experiencing partial to major outages with search and MyCludo on our EU servers. We are continuing to investigate the issues.</p>tag:status.cludo.com,2005:Incident/196878132024-01-12T14:37:22+01:002024-01-12T14:37:41+01:00Search API issues (EU)<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>14:37</var> CET</small><br><strong>Resolved</strong> - The incident has been resolved.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>14:29</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>14:20</var> CET</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>14:14</var> CET</small><br><strong>Update</strong> - We are continuing to investigate the issue and are working hard to get things back up and running as soon as possible.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>13:43</var> CET</small><br><strong>Investigating</strong> - We are currently seeing issues with search in the EU, affecting access to the MyCludo platform as well. We are investigating the issue.</p>tag:status.cludo.com,2005:Incident/196668512024-01-10T17:39:47+01:002024-01-10T17:48:17+01:00Crawler logs down (US only)<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>17:39</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved. Crawler logs are restored.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>12:54</var> CET</small><br><strong>Investigating</strong> - Crawler logs are currently down on US, meaning that it is not currently possible to view crawler logs in MyCludo. This does not affect actual crawling. We are currently investigating the issue.</p>tag:status.cludo.com,2005:Incident/196106762024-01-02T04:00:00+01:002024-01-04T11:48:46+01:00Partial to major search outage on US cluster<p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>04:00</var> CET</small><br><strong>Resolved</strong> - Our services in the US experienced a partial to major outage from around 9:00 AM to 1:00 PM CDT due to an infrastructure issue. All systems were restarted and are now back online and fully operational.</p>tag:status.cludo.com,2005:Incident/191813032023-11-21T11:08:17+01:002023-11-21T11:08:17+01:00Temporarily missing analytics data<p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>11:08</var> CET</small><br><strong>Resolved</strong> - The data has been backfilled.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>09:00</var> CET</small><br><strong>Identified</strong> - We have identified analytics events that have been dropped prior to being stored in the database that drives the MyCludo dashboard and analytics newsletters.<br /><br />We expect the events to be backfilled in the analytics database by end of day November 21, 2023.</p>tag:status.cludo.com,2005:Incident/190991922023-11-12T07:45:16+01:002023-11-13T08:58:58+01:00Search outage on US cluster<p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>07:45</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>06:55</var> CET</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>06:45</var> CET</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>03:40</var> CET</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cludo.com,2005:Incident/190566902023-11-08T13:30:16+01:002023-11-08T13:51:31+01:00Partial Search Outage on US cluster<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>13:30</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>13:00</var> CET</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cludo.com,2005:Incident/189018202023-10-23T19:32:06+02:002023-10-24T14:21:09+02:00Downtime for search in EU<p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>19:32</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>19:18</var> CEST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>19:17</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>19:10</var> CEST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.cludo.com,2005:Incident/186968922023-10-05T11:05:46+02:002023-10-11T17:45:34+02:00Search outage on EU cluster<p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>11:05</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>10:50</var> CEST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>10:35</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cludo.com,2005:Incident/186311892023-09-28T04:00:33+02:002023-09-28T13:45:06+02:00Degraded performance on US cluster<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>04:00</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>00:00</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cludo.com,2005:Incident/184311802023-09-07T23:30:05+02:002023-09-08T08:26:57+02:00Search outage<p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>23:30</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>23:15</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>23:00</var> CEST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>20:30</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cludo.com,2005:Incident/183845032023-09-03T17:51:00+02:002023-09-04T17:53:40+02:00Reduced crawling capacity for full crawls on EU<p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>17:51</var> CEST</small><br><strong>Resolved</strong> - Crawling capacity was significantly reduced due to networking issues September 2 - September 4. During that timeframe, page additions or updates may not have made it into the index during full crawl. Crawlers are fully operational now.</p>tag:status.cludo.com,2005:Incident/180865072023-08-08T17:00:00+02:002023-08-08T17:59:43+02:00Degraded search performance<p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>17:00</var> CEST</small><br><strong>Resolved</strong> - We had a partial outage of a node in a cluster that affected search performance for about 30 minutes. The cause has been identified and the issue is resolved.</p>tag:status.cludo.com,2005:Incident/158289702023-01-12T17:23:14+01:002023-01-12T17:23:15+01:00Investigating degraded search performance<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>17:23</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>17:23</var> CET</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>16:59</var> CET</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cludo.com,2005:Incident/126923782022-11-02T10:00:00+01:002022-11-02T10:45:18+01:00Faulty API server<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>10:00</var> CET</small><br><strong>Resolved</strong> - We experienced an issue with an API server that had a corrupt connection to one of its dependencies and resulted in dropping most requests that went through it. It represented a fraction of all search requests. We've identified the issue and are working to minimize the risk going forward.</p>tag:status.cludo.com,2005:Incident/113275012022-09-30T16:09:48+02:002022-09-30T16:09:48+02:00Partial search out in US<p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>16:09</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>15:45</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>