tag:www.sanity-status.com,2005:/historySanity Status - Incident History2024-03-28T16:47:14+01:00Sanitytag:www.sanity-status.com,2005:Incident/202453832024-03-14T17:44:22+01:002024-03-14T18:15:12+01:00Elevated API error rates for some customers<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>17:44</var> CET</small><br><strong>Resolved</strong> - Between 13:54 and 14:18 UTC, we experienced an elevated level of API errors (502 status code) served to some of our customers. The issue was identified, a fix was implemented, and we continue to monitor.</p>tag:www.sanity-status.com,2005:Incident/198827572024-02-01T10:59:41+01:002024-02-01T10:59:41+01:00Manage for Plans is unavailable<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>10:59</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>10:24</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>09:58</var> CET</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>09:44</var> CET</small><br><strong>Investigating</strong> - We are currently investigating an issue with Plans being inaccessible on sanity.io/manage. Customers are currently unable to view and update plans. This issue is affecting both upgrades and downgrades of plans.</p>tag:www.sanity-status.com,2005:Incident/198254082024-01-25T05:30:22+01:002024-01-25T05:45:14+01:00Some queries failing with 5xx status codes<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>05:30</var> CET</small><br><strong>Resolved</strong> - Between 2024-01-24 at 22:00 and 2024-01-25 at 04:10 UTC, we observed a small number of 504 and 524 errors were served for a small number of query and graphql requests. We have resolved the underlying issue and are no longer observing those errors.</p>tag:www.sanity-status.com,2005:Incident/197438252024-01-17T23:21:22+01:002024-01-19T14:56:46+01:00Sanity.io marketing domains reporting down - core services unaffected<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>23:21</var> CET</small><br><strong>Resolved</strong> - Vercel has reported that error rates have stabilized and is continuing to monitor the situation.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>23:12</var> CET</small><br><strong>Update</strong> - Reports indicate multiple users have reported encountering 500 errors when accessing sanity.io. Vercel has reported an incident on their status page of elevated 500 error rates on edge requests. We have updated the status page to inform users about the ongoing issue.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>23:11</var> CET</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>23:07</var> CET</small><br><strong>Investigating</strong> - We are currently investigating reports of domains on sanity.io returning 500 internal server errors.</p>tag:www.sanity-status.com,2005:Incident/197370682024-01-17T07:00:00+01:002024-01-17T12:49:00+01:00Studio login issues for European users seeing telemetry data prompt<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>07:00</var> CET</small><br><strong>Resolved</strong> - Between 06:03 UTC and 10:58 UTC, European users experienced difficulties logging in to their Studios because of a telemetry data prompt getting them stuck in a loop. We have rolled back an update and the issue is now resolved. We continue investigating the root cause and apologise for the impact this issue may have had on your operations and business.</p>tag:www.sanity-status.com,2005:Incident/195968282024-01-02T23:30:00+01:002024-01-03T00:41:30+01:00Some cross-dataset references returning null<p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>23:30</var> CET</small><br><strong>Resolved</strong> - Between 16:02 and 23:14 UTC, some customers may have experienced cross-dataset references that were returning null rather than the expected data. This regression was introduced by a changed query in the datastore codebase; customer data was not impacted and remained accessible via direct query. Normal reference functionality was unaffected. We regret this may have resulted in incomplete query results and are sorry for any impact this had on your operations or business.</p>tag:www.sanity-status.com,2005:Incident/195946202024-01-02T15:30:00+01:002024-01-02T18:42:03+01:00Issues accepting invitations to projects<p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>15:30</var> CET</small><br><strong>Resolved</strong> - Between 14:38 and 16:59 UTC, people experienced difficulties accepting project invitations, receiving an error message "Unable to accept invitation. An internal server error has occurred." A fix was implemented and the issue was resolved. People should now be able to accept their original invites. We continue monitoring closely and apologize for any impact this may have caused.</p>tag:www.sanity-status.com,2005:Incident/193507142023-12-05T17:00:00+01:002023-12-06T13:32:05+01:00Issues with Sanity's Vercel integration<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>17:00</var> CET</small><br><strong>Resolved</strong> - Between Dec 5, 15:47 UTC and Dec 6, 12:10 UTC, people were unable to use our Vercel integration. A fix was implemented and the issue is now resolved. We apologize for any impact this might have caused when integrating Sanity with your Vercel projects.</p>tag:www.sanity-status.com,2005:Incident/191176382023-11-14T14:21:00+01:002023-11-14T14:31:47+01:00Elevated API error rates for a subset of customers<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>14:21</var> CET</small><br><strong>Resolved</strong> - Between 13:06 and 13:20 UTC, we experienced an elevated level of API errors for some of our customers. The issue was identified, a fix was implemented, and we continue monitoring results.</p>tag:www.sanity-status.com,2005:Incident/190860362023-11-10T18:49:26+01:002023-11-10T18:49:26+01:00Datasets tab does not render the list of datasets for some customers<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>18:49</var> CET</small><br><strong>Resolved</strong> - This issue has been resolved.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>18:35</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>18:10</var> CET</small><br><strong>Identified</strong> - The issue has been identified and we are releasing a fix.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>17:54</var> CET</small><br><strong>Investigating</strong> - The Datasets tab in Manage is not rendering the list of datasets for some customers. This is limited to the Manage dashboard and does not impact the datasets themselves. We are currently investigating the issue.</p>tag:www.sanity-status.com,2005:Incident/190597752023-11-08T17:15:00+01:002023-11-08T20:27:41+01:00Elevated API and API CDN error rates for some customers<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>17:15</var> CET</small><br><strong>Resolved</strong> - Between 16:03 UTC and 16:15 UTC, we experienced an elevated level of API and API CDN errors for some of our customers. An upstream issue with a service provider was mitigated and we continue monitoring closely.</p>tag:www.sanity-status.com,2005:Incident/190586942023-11-08T16:27:42+01:002023-11-08T16:27:42+01:00Elevated failures for dataset Cloud Clone<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>16:27</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>16:19</var> CET</small><br><strong>Investigating</strong> - Dataset Cloud Clone is failing for some users. We're currently investigating this issue.</p>tag:www.sanity-status.com,2005:Incident/190456472023-11-07T13:30:00+01:002023-11-07T14:05:23+01:00Queries and mutations failing for some datasets<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>13:30</var> CET</small><br><strong>Resolved</strong> - Between 12:31 and 12:41 (UTC), queries and mutations failed for some datasets. We have identified and fixed the cause of this incident, and will continue to monitor API requests.</p>tag:www.sanity-status.com,2005:Incident/190343202023-11-06T15:49:15+01:002023-11-06T15:49:15+01:00Elevated failures for dataset Cloud Clone<p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>15:49</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>14:29</var> CET</small><br><strong>Monitoring</strong> - We have identified and fixed the cause of this issue. We're continuing to monitor Cloud Clone processes.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>13:47</var> CET</small><br><strong>Investigating</strong> - Dataset Cloud Clone is failing for some users. We're currently investigating this issue.</p>tag:www.sanity-status.com,2005:Incident/186422542023-09-29T15:04:51+02:002023-09-29T15:04:51+02:00Degraded performance and issues loading data on Manage<p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>15:04</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>14:59</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>14:53</var> CEST</small><br><strong>Investigating</strong> - Starting at 10:50 UTC, we are experiencing degraded performance and issues loading data on Manage. People may be unable to see and edit their plans. Plan data and billing are not impacted. We are currently investigating this incident.<br /><br />Services affected: Manage<br />Next update: 30 minutes</p>tag:www.sanity-status.com,2005:Incident/185021902023-09-14T10:30:00+02:002023-09-14T17:52:45+02:00Asset pipeline issue for some customers<p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>10:30</var> CEST</small><br><strong>Resolved</strong> - We experienced an issue with our asset pipeline that affected a small subset of customers. This issue may have impacted those trying to upload an asset for a subsequent time, which is a common occurrence during the dataset import process. The issue would have started at or shortly after 17:47 UTC on 2023-09-13 and was resolved at 08:26 UTC on 2023-09-14. We apologize for the impact this had and any backup processes that were disrupted.</p>tag:www.sanity-status.com,2005:Incident/183944092023-09-05T14:26:56+02:002023-09-06T04:21:15+02:00Global API CDN unavailable for 30 minutes<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>14:26</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>14:18</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>13:59</var> CEST</small><br><strong>Investigating</strong> - We are experiencing an elevated level of API errors for some of our customers and are currently investigating the issue.</p>tag:www.sanity-status.com,2005:Incident/178170072023-07-15T09:49:06+02:002023-07-15T09:49:07+02:00Scheduled Maintenance on July 15th<p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>09:49</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>09:24</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>09:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>21:21</var> CEST</small><br><strong>Scheduled</strong> - Sanity will perform essential maintenance on July 15, 2023, from 12:00 AM to 01:00 AM PDT, requiring a brief service disruption period.<br /><br />During this window, the following services will be impacted:<br />- Authentication and Sign-ups: unauthenticated users will not be able to sign up or log in to Manage, their Studio, or Sanity CLI via social login, username and password or SAML/SSO. Users and service tokens logged in before this window will have regular access.<br />- Project Management, Organization Management, User Management, and Dataset Settings: create, update, and delete operations will not work. All read operations will work.<br />- Integrations: users will not be able to activate integrations to Sanity Connect for Shopify and Vercel. Existing integrations will keep operating normally.<br /><br />Unimpacted services:<br />- Your Studio will continue to be operational for logged-in users.<br />- All content-related operations to the API endpoint will continue working. This includes creating, updating, querying, and receiving webhook and listener events.<br />- The Asset CDN will be operational.<br />- The API CDN will be operational.<br /><br />We apologize for any impact this may have on your operations and appreciate your understanding.</p>tag:www.sanity-status.com,2005:Incident/178229202023-07-11T15:21:20+02:002023-07-11T15:21:20+02:00Issues loading data affecting some datasets<p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>15:21</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>13:32</var> CEST</small><br><strong>Update</strong> - Datasets are accessible again, and we are continuing to monitor the situation.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>13:28</var> CEST</small><br><strong>Monitoring</strong> - Datasets are accessible again, and we are continuing to monitor the situation.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>12:33</var> CEST</small><br><strong>Investigating</strong> - Some customers have experienced issues loading data from their datasets. We are currently investigating this incident.</p>tag:www.sanity-status.com,2005:Incident/177097502023-06-28T04:00:00+02:002023-06-28T17:08:26+02:00Elevated API error rates for some customers<p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>04:00</var> CEST</small><br><strong>Resolved</strong> - From 01:59-02:22 UTC, we experienced an elevated level of API errors for some of our customers. We have implemented a fix and are monitoring results.</p>tag:www.sanity-status.com,2005:Incident/175756122023-06-15T00:10:41+02:002023-06-15T00:10:41+02:00Elevated API error rates for some customers<p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>00:10</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>23:02</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>22:16</var> CEST</small><br><strong>Investigating</strong> - We are experiencing an elevated level of API errors for some of our customers and are currently investigating the issue.</p>tag:www.sanity-status.com,2005:Incident/173039562023-05-19T12:55:00+02:002023-05-19T18:17:37+02:00Some GraphQL queries failing with 400 status code<p><small>May <var data-var='date'>19</var>, <var data-var='time'>12:55</var> CEST</small><br><strong>Resolved</strong> - For a period of approximately 55 minutes, some GraphQL queries failed with a 400 status code. This was caused by a misconfiguration in our GraphQL API service, which we rolled back and have since rectified.</p>tag:www.sanity-status.com,2005:Incident/171414402023-05-03T13:52:00+02:002023-05-04T11:12:55+02:00Dataset aliases not working for images requested from the CDN<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>13:52</var> CEST</small><br><strong>Resolved</strong> - Some images requested from the Sanity CDN using a dataset alias incorrectly resulted in a 404 response. This incident began at approximately 11:52 UTC and was resolved at approximately 15:20 UTC. This incident was limited to requests using a dataset alias.</p>tag:www.sanity-status.com,2005:Incident/168550652023-04-13T23:08:28+02:002023-04-13T23:08:28+02:00Insufficient user permissions in Studio for some projects<p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>23:08</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>21:26</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>21:11</var> CEST</small><br><strong>Identified</strong> - We have identified an issue where some users with default roles are not being provided with sufficient permissions in the Studio, which is preventing document creation and edits. A fix is forthcoming.</p>tag:www.sanity-status.com,2005:Incident/165893532023-03-21T11:54:48+01:002023-03-21T12:25:57+01:00Elevated API error rates for some customers<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>11:54</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>11:20</var> CET</small><br><strong>Investigating</strong> - We are experiencing an elevated level of API errors for some of our customers and are currently investigating the issue.</p>