Incidents | Square Cloud Incidents reported on status page for Square Cloud https://status.squarecloud.app/ https://d1lppblt9t2x15.cloudfront.net/logos/1921ca315133941a7235c57c3138a4ed.png Incidents | Square Cloud https://status.squarecloud.app/ en Network instability (Telekom Italia) https://status.squarecloud.app/incident/562558 Wed, 14 May 2025 18:07:00 -0000 https://status.squarecloud.app/incident/562558#c9e155141e762cff4023fa423d96f3e9cdc9e2c55072289b053ead40535f9f6f Our network team has received a response from Telekom Italia Brazil informing us that the issue has been resolved. Network instability (Telekom Italia) https://status.squarecloud.app/incident/562558 Tue, 13 May 2025 15:37:00 -0000 https://status.squarecloud.app/incident/562558#b77a354d5b904b33e9c6c365b053fbc83e96e962a28177d14ae1f90dd3c7459d Instability apparently resolved, we have not yet received a response from Telekom Italia. Waiting for further updates. Network instability (Telekom Italia) https://status.squarecloud.app/incident/562558 Tue, 13 May 2025 14:56:00 -0000 https://status.squarecloud.app/incident/562558#babc7e8b65e1249b5232ee2936c1843583b533f02f57e399974b84fa8e41c160 Our Network Team contacted Telekom Italia Brazil (TIM) seeking feedback on when the instability will be resolved. We reiterate that the instability is not from Square but rather from connections coming from Brazil. We also detected instabilities from Vivo (but no response so far). Network instability (Telekom Italia) https://status.squarecloud.app/incident/562558 Tue, 13 May 2025 14:45:00 -0000 https://status.squarecloud.app/incident/562558#076456f68119d002b44e3431572a531b7a994932174f0447a0efcdc9cf76cc93 Instability appears to be isolated to cases in LATAM, all checks in Europe, Asia and North America show no failures. Connections coming from Latin America, we detected instabilities via the Vivo and Tim Fiber network, we are looking for some status update to provide here. Network instability (Telekom Italia) https://status.squarecloud.app/incident/562558 Tue, 13 May 2025 14:28:00 -0000 https://status.squarecloud.app/incident/562558#e0b41aa8909c8b33fb53fa2342354771a3b6d3c41a30f3ef5985a407af4de1b5 We have detected an instability in our network that affects all clusters, we are investigating the cause of the situation. Frontend deploys https://status.squarecloud.app/incident/555976 Sat, 03 May 2025 16:43:00 -0000 https://status.squarecloud.app/incident/555976#bce60b93a813b09f4101eb8a1e9b78eb9a60563fb54bdf5929f3661b3deb2435 The recent infrastructure instability has been mitigated. We have doubled the processing capacity of our workers and believe the issue affecting the provisioning of new workers has been partially resolved. We are continuing to monitor the situation closely. Frontend deploys https://status.squarecloud.app/incident/555976 Sat, 03 May 2025 16:32:00 -0000 https://status.squarecloud.app/incident/555976#05f1f57e6e01285e3577140ed6fa989700a56bdca3ca8dc8619063a4f62e43d9 We're currently experiencing instability in Square Cloud's frontend provisioning service due to an ongoing attack. Our automatic scaling mechanism is not being triggered as expected, and we're actively investigating the root cause to restore stability as quickly as possible. Deploys https://status.squarecloud.app/incident/554556 Wed, 30 Apr 2025 19:47:00 -0000 https://status.squarecloud.app/incident/554556#f84d8e83e034b07f93996b9bc1feba15369a456277bfa3fdb3b6baae1cf1068e Failure resolved, we will continue monitoring the infrastructure until all traffic is normalized. One of our load balancers failed due to a Redis break change, which was not previously communicated and due to a lack of testing in this niche (inside of the load balancer), it ended up not being detected. Sorry. 🫠 Deploys https://status.squarecloud.app/incident/554556 Wed, 30 Apr 2025 19:34:00 -0000 https://status.squarecloud.app/incident/554556#6ddb91fdabbb403501471c360e12e58405593d9c4fbfd11561494a4cb10f131a We have identified instability in sending new releases, especially involving submissions from our API coming from South America, we are investigating the cause. HE.net downtime. https://status.squarecloud.app/incident/548239 Sun, 20 Apr 2025 21:17:00 -0000 https://status.squarecloud.app/incident/548239#4ef605c72fef95ca99a347958eb52a7f10abf2c61f766330d2aeb136e5af5c35 Our network team has reported that the recent network instability appears to be resolved. Affected traffic was successfully rerouted through our redundant connections during the incident. We will continue to actively monitor the situation to ensure full stability. HE.net downtime. https://status.squarecloud.app/incident/548239 Sun, 20 Apr 2025 21:09:00 -0000 https://status.squarecloud.app/incident/548239#ce9493b78c04a90360ad28ab8705ebc6112dacea586a9564945637e42e08f541 We have detected a critical instability in HE. Our network team is already performing a full network swap to ensure complete service continuity. Please note that this is a network instability, not a downtime. Website degraded https://status.squarecloud.app/incident/532430 Fri, 21 Mar 2025 22:23:00 -0000 https://status.squarecloud.app/incident/532430#bf16deb9ee89e357d73826da9d73ff3a868cbd8f8f0cb3ade0d13af4acc6dd52 Our website is now fully operational again. However, the recent downtime was caused by an unfortunate timing of a failed front-end update alongside an exclusive-level attack targeting our servers, mainly affecting routes in LATAM and Europe. In total, in 17 minutes we mitigated 2.16 billion requests, ensuring platform security and stability. We appreciate your patience and remain committed to strengthening our infrastructure. Website degraded https://status.squarecloud.app/incident/532430 Fri, 21 Mar 2025 22:13:00 -0000 https://status.squarecloud.app/incident/532430#d80e0ac8b9bdf61f63fd966b5cbf6c42135a8bc83a9e5329eaec825669a73ee1 We are mitigating an exclusive level attack on our servers generally affecting our website routes available in LATAM and EUROPE, on the other hand an update to our website (front-end) failed, causing a longer delay in making the website available. Maintenance FL-MICRON-7 and 8. https://status.squarecloud.app/incident/455953 Mon, 04 Nov 2024 22:47:00 -0000 https://status.squarecloud.app/incident/455953#28597105ce8df766ef592f759a8efaf3e706318eabc5beff87f68163a7df1d7d Cluster FL-MICRON-8 released for normal operation, we apologize for the inconvenience, any application that has been going offline will start within 1 hour. Maintenance FL-MICRON-7 and 8. https://status.squarecloud.app/incident/455953 Mon, 04 Nov 2024 22:41:00 -0000 https://status.squarecloud.app/incident/455953#ca8da0974e5bbaf1857c76dac4ddccd2297cb6e913a66bbfd370b4bef3623b26 We have successfully replaced a driver on the FL-MICRON-8, and our technical team has resolved the issue. The cluster will be fully operational again in a few minutes. Thank you for your patience. Maintenance FL-MICRON-7 and 8. https://status.squarecloud.app/incident/455953 Mon, 04 Nov 2024 22:00:00 -0000 https://status.squarecloud.app/incident/455953#d59d9c8c4a993ca3df9f7eb63190b03e463b72851d5292bc7cde23122bf99730 We are currently experiencing instability across multiple XFS-based file systems on our FL-MICRON-8 drives. Our technical team has been engaged, and the local data center team is assisting with verification and resolution as quickly as possible. Maintenance FL-MICRON-7 and 8. https://status.squarecloud.app/incident/455953 Mon, 04 Nov 2024 21:12:00 -0000 https://status.squarecloud.app/incident/455953#129661640bd9c19d0326b79e47e7f9e21fb414b02de9e2e022b1fa0c20d30ec2 Cluster FL-MICRON-7 released for normal operation, we apologize for the inconvenience, any application that has been going offline will start within 1 hour. Maintenance FL-MICRON-7 and 8. https://status.squarecloud.app/incident/455953 Mon, 04 Nov 2024 20:37:00 -0000 https://status.squarecloud.app/incident/455953#adf7ba9551aca5a4c27a2d2a23eb0464dde4c89b3413b3ad2cb1f3222e5eb638 We are implementing enhancements to stabilize our clusters following the identification of a vulnerability that impacted performance. We appreciate your understanding and cooperation during this time. Network unstable https://status.squarecloud.app/incident/422255 Sat, 31 Aug 2024 17:39:00 -0000 https://status.squarecloud.app/incident/422255#215f8809ace5b0afd09631ff9aff34b87e2ea91086a8208b700d473f4798c46c We have identified that the instability has been resolved, but we will continue to monitor the situation. Network unstable https://status.squarecloud.app/incident/422255 Sat, 31 Aug 2024 17:39:00 -0000 https://status.squarecloud.app/incident/422255#215f8809ace5b0afd09631ff9aff34b87e2ea91086a8208b700d473f4798c46c We have identified that the instability has been resolved, but we will continue to monitor the situation. Network unstable https://status.squarecloud.app/incident/422255 Sat, 31 Aug 2024 17:11:00 -0000 https://status.squarecloud.app/incident/422255#a26eeadb3ad5809e2ce1ddc6be27e4a3240383cc80a97c9bf7f88bbdc243d104 We detect that some connections to our services then present instabilities and packet loss. This problem is related to the TPA network. We monitor the situation. Network unstable https://status.squarecloud.app/incident/422255 Sat, 31 Aug 2024 17:11:00 -0000 https://status.squarecloud.app/incident/422255#a26eeadb3ad5809e2ce1ddc6be27e4a3240383cc80a97c9bf7f88bbdc243d104 We detect that some connections to our services then present instabilities and packet loss. This problem is related to the TPA network. We monitor the situation. Unstable Blogs and Documentation. https://status.squarecloud.app/incident/410454 Wed, 07 Aug 2024 20:11:00 -0000 https://status.squarecloud.app/incident/410454#8a7c69b41e6ce52db35035e4f8f617045a9f13ae42dc0c9f7ee866c885b2fc49 We are pleased to inform you that Mintlify has resolved the issues they were experiencing. The investigation has concluded, and their services are now fully operational. We appreciate your patience and understanding during this time. Please note that the issues were not related to Square Cloud or our infrastructure. Unstable Blogs and Documentation. https://status.squarecloud.app/incident/410454 Wed, 07 Aug 2024 20:11:00 -0000 https://status.squarecloud.app/incident/410454#8a7c69b41e6ce52db35035e4f8f617045a9f13ae42dc0c9f7ee866c885b2fc49 We are pleased to inform you that Mintlify has resolved the issues they were experiencing. The investigation has concluded, and their services are now fully operational. We appreciate your patience and understanding during this time. Please note that the issues were not related to Square Cloud or our infrastructure. Unstable Blogs and Documentation. https://status.squarecloud.app/incident/410454 Wed, 07 Aug 2024 19:40:00 -0000 https://status.squarecloud.app/incident/410454#440eda320ec14c96956760ff525ea696ed39d1c8b043e0206ac6281a0ab1ddd1 We have detected issues with our trading partner Mintlify and will be conducting an investigation. Please note that this is not an issue with Square Cloud or our infrastructure. Unstable Blogs and Documentation. https://status.squarecloud.app/incident/410454 Wed, 07 Aug 2024 19:40:00 -0000 https://status.squarecloud.app/incident/410454#440eda320ec14c96956760ff525ea696ed39d1c8b043e0206ac6281a0ab1ddd1 We have detected issues with our trading partner Mintlify and will be conducting an investigation. Please note that this is not an issue with Square Cloud or our infrastructure. Dashboard Maintenance https://status.squarecloud.app/incident/398562 Mon, 15 Jul 2024 20:42:54 +0000 https://status.squarecloud.app/incident/398562#bc8589d913bc71f4f5e1cc38b583b725c6ce07e632f946dc78a39eaca2912a78 Maintenance completed Dashboard Maintenance https://status.squarecloud.app/incident/398562 Mon, 15 Jul 2024 20:42:54 -0000 https://status.squarecloud.app/incident/398562#d2ad806d409127a9de6bfdf798af97e2aee4150c1abf7ccd7f4a345a6efa1840 Dashboard is currently unavailable for small percentage of our users. We are scaling the workers on our website, however, we took advantage of the moment to update our infrastructure responsible for our dashboard. Dashboard Maintenance https://status.squarecloud.app/incident/398562 Mon, 15 Jul 2024 20:40:00 -0000 https://status.squarecloud.app/incident/398562#d2ad806d409127a9de6bfdf798af97e2aee4150c1abf7ccd7f4a345a6efa1840 Dashboard is currently unavailable for small percentage of our users. We are scaling the workers on our website, however, we took advantage of the moment to update our infrastructure responsible for our dashboard. Storage Maintenance https://status.squarecloud.app/incident/398164 Mon, 15 Jul 2024 00:59:14 +0000 https://status.squarecloud.app/incident/398164#4cc7dfe459c544115b7aaa0bbd5a4f02ba8bf495e62461b3109527734dda6dd1 Maintenance completed Storage Maintenance https://status.squarecloud.app/incident/398164 Mon, 15 Jul 2024 00:59:14 -0000 https://status.squarecloud.app/incident/398164#01211126bc0802b940a9721f18a1d909ebcda84634995f466301f7a8eb343ff6 Due to scheduled maintenance at the Miami datacenter, which we use as a bridge to distribute content mainly to Brazil and Europe, our CDN and Blob services may be affected. During this maintenance period, the creation of new backups will be suspended, and the uploading of new files may be impacted. Additionally, the listing of existing backups may be partially affected. Please note that this maintenance is being conducted by a commercial partner of Square Cloud, not by Square Cloud itself. We apologize for any inconvenience and appreciate your understanding. Storage Maintenance https://status.squarecloud.app/incident/398164 Mon, 15 Jul 2024 00:59:14 +0000 https://status.squarecloud.app/incident/398164#4cc7dfe459c544115b7aaa0bbd5a4f02ba8bf495e62461b3109527734dda6dd1 Maintenance completed Storage Maintenance https://status.squarecloud.app/incident/398164 Mon, 15 Jul 2024 00:59:14 -0000 https://status.squarecloud.app/incident/398164#01211126bc0802b940a9721f18a1d909ebcda84634995f466301f7a8eb343ff6 Due to scheduled maintenance at the Miami datacenter, which we use as a bridge to distribute content mainly to Brazil and Europe, our CDN and Blob services may be affected. During this maintenance period, the creation of new backups will be suspended, and the uploading of new files may be impacted. Additionally, the listing of existing backups may be partially affected. Please note that this maintenance is being conducted by a commercial partner of Square Cloud, not by Square Cloud itself. We apologize for any inconvenience and appreciate your understanding. Storage Maintenance https://status.squarecloud.app/incident/398164 Mon, 15 Jul 2024 00:59:14 +0000 https://status.squarecloud.app/incident/398164#4cc7dfe459c544115b7aaa0bbd5a4f02ba8bf495e62461b3109527734dda6dd1 Maintenance completed Storage Maintenance https://status.squarecloud.app/incident/398164 Mon, 15 Jul 2024 00:59:14 -0000 https://status.squarecloud.app/incident/398164#01211126bc0802b940a9721f18a1d909ebcda84634995f466301f7a8eb343ff6 Due to scheduled maintenance at the Miami datacenter, which we use as a bridge to distribute content mainly to Brazil and Europe, our CDN and Blob services may be affected. During this maintenance period, the creation of new backups will be suspended, and the uploading of new files may be impacted. Additionally, the listing of existing backups may be partially affected. Please note that this maintenance is being conducted by a commercial partner of Square Cloud, not by Square Cloud itself. We apologize for any inconvenience and appreciate your understanding. Storage Maintenance https://status.squarecloud.app/incident/398164 Mon, 15 Jul 2024 00:30:00 -0000 https://status.squarecloud.app/incident/398164#01211126bc0802b940a9721f18a1d909ebcda84634995f466301f7a8eb343ff6 Due to scheduled maintenance at the Miami datacenter, which we use as a bridge to distribute content mainly to Brazil and Europe, our CDN and Blob services may be affected. During this maintenance period, the creation of new backups will be suspended, and the uploading of new files may be impacted. Additionally, the listing of existing backups may be partially affected. Please note that this maintenance is being conducted by a commercial partner of Square Cloud, not by Square Cloud itself. We apologize for any inconvenience and appreciate your understanding. Storage Maintenance https://status.squarecloud.app/incident/398164 Mon, 15 Jul 2024 00:30:00 -0000 https://status.squarecloud.app/incident/398164#01211126bc0802b940a9721f18a1d909ebcda84634995f466301f7a8eb343ff6 Due to scheduled maintenance at the Miami datacenter, which we use as a bridge to distribute content mainly to Brazil and Europe, our CDN and Blob services may be affected. During this maintenance period, the creation of new backups will be suspended, and the uploading of new files may be impacted. Additionally, the listing of existing backups may be partially affected. Please note that this maintenance is being conducted by a commercial partner of Square Cloud, not by Square Cloud itself. We apologize for any inconvenience and appreciate your understanding. Storage Maintenance https://status.squarecloud.app/incident/398164 Mon, 15 Jul 2024 00:30:00 -0000 https://status.squarecloud.app/incident/398164#01211126bc0802b940a9721f18a1d909ebcda84634995f466301f7a8eb343ff6 Due to scheduled maintenance at the Miami datacenter, which we use as a bridge to distribute content mainly to Brazil and Europe, our CDN and Blob services may be affected. During this maintenance period, the creation of new backups will be suspended, and the uploading of new files may be impacted. Additionally, the listing of existing backups may be partially affected. Please note that this maintenance is being conducted by a commercial partner of Square Cloud, not by Square Cloud itself. We apologize for any inconvenience and appreciate your understanding. Deployments are degraded. https://status.squarecloud.app/incident/372749 Wed, 22 May 2024 09:33:00 -0000 https://status.squarecloud.app/incident/372749#6c66393cde80d29661ad2ba58e1d7af40b7054fe4cab8d3d051d3fd545262bb6 The issue has been resolved. We apologize for any inconvenience caused. Deployments are degraded. https://status.squarecloud.app/incident/372749 Wed, 22 May 2024 09:33:00 -0000 https://status.squarecloud.app/incident/372749#6c66393cde80d29661ad2ba58e1d7af40b7054fe4cab8d3d051d3fd545262bb6 The issue has been resolved. We apologize for any inconvenience caused. Deployments are degraded. https://status.squarecloud.app/incident/372749 Wed, 22 May 2024 01:27:00 -0000 https://status.squarecloud.app/incident/372749#7bd4f00a4b2c689ffc9fc64ef444583b03d2c9dba8b288ff103d5f057360c1b4 We have once again detected increased latency and slight instability in our deployments due to issues with a third-party provider. We apologize for any inconvenience this may have caused and are actively seeking alternatives and new partners to ensure a more stable and efficient service in the future. Deployments are degraded. https://status.squarecloud.app/incident/372749 Wed, 22 May 2024 01:27:00 -0000 https://status.squarecloud.app/incident/372749#7bd4f00a4b2c689ffc9fc64ef444583b03d2c9dba8b288ff103d5f057360c1b4 We have once again detected increased latency and slight instability in our deployments due to issues with a third-party provider. We apologize for any inconvenience this may have caused and are actively seeking alternatives and new partners to ensure a more stable and efficient service in the future. Proxy degraded. https://status.squarecloud.app/incident/360819 Thu, 25 Apr 2024 20:25:00 -0000 https://status.squarecloud.app/incident/360819#b5f746b7c5eee7eb34815d0906a17256f6c0550744019048dadc5a6e2c7d8673 Network instability resolved, duration of about 7 minutes. We will continue to monitor the network. Proxy degraded. https://status.squarecloud.app/incident/360819 Thu, 25 Apr 2024 20:25:00 -0000 https://status.squarecloud.app/incident/360819#b5f746b7c5eee7eb34815d0906a17256f6c0550744019048dadc5a6e2c7d8673 Network instability resolved, duration of about 7 minutes. We will continue to monitor the network. Proxy degraded. https://status.squarecloud.app/incident/360819 Thu, 25 Apr 2024 20:23:00 -0000 https://status.squarecloud.app/incident/360819#781321cad230f5e6c75360de3396f10d3a59456f91f7b26842b7492cd29fb275 Network problem identified, instability is due to a problem in Hivelocity, our datacenter. Packet lost of around 80%. We apologize for the instability. Proxy degraded. https://status.squarecloud.app/incident/360819 Thu, 25 Apr 2024 20:23:00 -0000 https://status.squarecloud.app/incident/360819#781321cad230f5e6c75360de3396f10d3a59456f91f7b26842b7492cd29fb275 Network problem identified, instability is due to a problem in Hivelocity, our datacenter. Packet lost of around 80%. We apologize for the instability. Proxy degraded. https://status.squarecloud.app/incident/360819 Thu, 25 Apr 2024 20:19:00 -0000 https://status.squarecloud.app/incident/360819#6c7dc7d29b15a8c04ad0541f51fda0a35f6222204d4f362e1437222914528536 We have escalated the instability to all services that depend on our main proxy, which have affected connectivity. Network team has already been notified. Proxy degraded. https://status.squarecloud.app/incident/360819 Thu, 25 Apr 2024 20:19:00 -0000 https://status.squarecloud.app/incident/360819#6c7dc7d29b15a8c04ad0541f51fda0a35f6222204d4f362e1437222914528536 We have escalated the instability to all services that depend on our main proxy, which have affected connectivity. Network team has already been notified. Proxy degraded. https://status.squarecloud.app/incident/360819 Thu, 25 Apr 2024 20:16:00 -0000 https://status.squarecloud.app/incident/360819#92cafd4ba07dcd20961ddf95a9058b2dbf2501f1f81f597ff9c1a61835516d84 Connections to Miami are aborting, affecting all connectivity in Brazil and similar regions, we are trying to identify what happened internally, as the loadbalancer is not distributing the load correctly.