Incidents reported on status page for Fedecan https://status.fedecan.ca/ en https://status.fedecan.ca/ Fri, 16 May 2025 08:51:09 +0000 https://status.fedecan.ca/#586c9b73f7e7fd9d66c5e39f12ce3fd21b79ebfd76d1b30167abdceb764c9895 pixelfed.ca recovered https://status.fedecan.ca/ Fri, 16 May 2025 08:48:08 +0000 https://status.fedecan.ca/#586c9b73f7e7fd9d66c5e39f12ce3fd21b79ebfd76d1b30167abdceb764c9895 pixelfed.ca went down https://status.fedecan.ca/incident/541995 Tue, 08 Apr 2025 17:00:09 +0000 https://status.fedecan.ca/incident/541995#577036e9051987229340cd92bc47ec00ee8a38d84f69aa8e2373f37d8cbda017 Maintenance completed https://status.fedecan.ca/ Tue, 08 Apr 2025 16:48:53 +0000 https://status.fedecan.ca/#a8a1ae91228e7f608e18dcb846a924fb74541b24273b8b7d614e075da5963a7c lemmy.ca recovered https://status.fedecan.ca/ Tue, 08 Apr 2025 16:34:38 +0000 https://status.fedecan.ca/#a8a1ae91228e7f608e18dcb846a924fb74541b24273b8b7d614e075da5963a7c lemmy.ca went down https://status.fedecan.ca/incident/541995 Tue, 08 Apr 2025 16:30:09 -0000 https://status.fedecan.ca/incident/541995#509305b82f70c551b6804a38bb5cbb95048ad0953788081099f0f0b671327230 Upgrading to 0.19.11 https://status.fedecan.ca/ Mon, 31 Mar 2025 08:26:47 +0000 https://status.fedecan.ca/#b70831e6e4abced310fbb270586d956b0e8d27e78f19ac5616a0ae2abcf483ce lemmy.ca recovered https://status.fedecan.ca/ Mon, 31 Mar 2025 08:23:49 +0000 https://status.fedecan.ca/#b70831e6e4abced310fbb270586d956b0e8d27e78f19ac5616a0ae2abcf483ce lemmy.ca went down https://status.fedecan.ca/ Sun, 23 Mar 2025 23:38:54 +0000 https://status.fedecan.ca/#db008132eab14208c1894871d1e293f022256b15e8ddad63db1133fae49f5e0d lemmy.ca recovered https://status.fedecan.ca/ Sun, 23 Mar 2025 23:35:58 +0000 https://status.fedecan.ca/#db008132eab14208c1894871d1e293f022256b15e8ddad63db1133fae49f5e0d lemmy.ca went down https://status.fedecan.ca/ Sat, 22 Mar 2025 01:50:22 +0000 https://status.fedecan.ca/#405c336fd48495dd34fea5a460890a58b3eb8f66702daa4708dd3c2c1312c573 lemmy.ca recovered https://status.fedecan.ca/ Sat, 22 Mar 2025 01:47:55 +0000 https://status.fedecan.ca/#405c336fd48495dd34fea5a460890a58b3eb8f66702daa4708dd3c2c1312c573 lemmy.ca went down https://status.fedecan.ca/ Sat, 15 Mar 2025 21:56:46 +0000 https://status.fedecan.ca/#ea44cd8120afe0f08e0b99bc3808a277c4279de336481f6bdcfc518a3e552458 lemmy.ca recovered https://status.fedecan.ca/ Sat, 15 Mar 2025 21:53:47 +0000 https://status.fedecan.ca/#ea44cd8120afe0f08e0b99bc3808a277c4279de336481f6bdcfc518a3e552458 lemmy.ca went down https://status.fedecan.ca/incident/502669 Thu, 30 Jan 2025 20:00:43 +0000 https://status.fedecan.ca/incident/502669#aa392bab86c301242f5045d4f08d4c0869dfd80d7368945bfb26608e084df299 Maintenance completed https://status.fedecan.ca/incident/502669 Thu, 30 Jan 2025 18:00:43 -0000 https://status.fedecan.ca/incident/502669#2662c3f20e877017c24e24c9410a5ec4b37cdb9bf3a1a0d20b40f7cb1f5b5334 We'll be offline for about 2 hours as we migrate pictrs from it's internal sled database, over to our postgres deployment https://status.fedecan.ca/incident/502668 Wed, 29 Jan 2025 20:00:58 +0000 https://status.fedecan.ca/incident/502668#6c96183bf7a894b2c07dc7f83eb6163552442e84b86ed302e2241c3dc8de1f14 Maintenance completed https://status.fedecan.ca/incident/502668 Wed, 29 Jan 2025 18:00:58 -0000 https://status.fedecan.ca/incident/502668#c927c18c0c838323b9ef0fa5333ab14b5f79dd59e73d466bd952e9c8803b088b We'll be offline while we migrate from OVH to our new dedicated hardware. https://status.fedecan.ca/incident/492702 Wed, 08 Jan 2025 18:05:34 +0000 https://status.fedecan.ca/incident/492702#99d9fa89f3fa009d87955ceb17739f9fb7c37ac1a2988d5df3f3c4d46b806e56 Maintenance completed https://status.fedecan.ca/incident/492702 Wed, 08 Jan 2025 18:00:34 -0000 https://status.fedecan.ca/incident/492702#e8ea47190cb6725d6b2931dc29fa625ed208b6d153518cc1eda1cadc476b59fe Taking the site down for about 5 minutes, so that we can get a consistent copy of our databases (postgres + pict-rs sled). https://status.fedecan.ca/incident/492083 Tue, 07 Jan 2025 16:42:00 -0000 https://status.fedecan.ca/incident/492083#31c0260d5d385312faeec86b3b5cfea4c1cf2229beec7737c131d39a87fe988a We're back again.... https://status.fedecan.ca/incident/492083 Tue, 07 Jan 2025 16:36:00 -0000 https://status.fedecan.ca/incident/492083#37572d7a6db7a7bc4f40e8abd7b8e4412ab1fa1e7e5e6fb071fa4bd4ca4856df They seem to be replacing our power supply again https://status.fedecan.ca/incident/492083 Tue, 07 Jan 2025 16:11:00 -0000 https://status.fedecan.ca/incident/492083#21fa02250e03b9f526535b689e357646c88dfcaadc6c1f2c1af56fa76a05deaa Ahhhh the server is down again. Talking to OVH to get us back up ASAP. https://status.fedecan.ca/incident/491086 Mon, 06 Jan 2025 20:21:00 -0000 https://status.fedecan.ca/incident/491086#a96b5510f6d9228eea378143b3926ef84af0199b1b138e9707ad1ba247dc20d4 We're back! https://lemmy.ca/post/36540245 https://status.fedecan.ca/incident/491086 Mon, 06 Jan 2025 19:18:00 -0000 https://status.fedecan.ca/incident/491086#c720b921cfd70034c56fcc2a7e80b703e26245e8fd8199878ee76adb0038e034 We're still waiting on OVH to repair the hardware, the datacenter tech has been working on it for about 3 hours now. Hopefully back soon! https://status.fedecan.ca/incident/491086 Sun, 05 Jan 2025 18:19:00 -0000 https://status.fedecan.ca/incident/491086#cae41b10df14806c503c9c5990f7580d72d27ccbced0a234ba6e45eacc8a2a7c Jan 5th 10:19am PST - Our server appears to have had a physical hardware failure. We have a support ticket open with OVH, but no ETA on repairs at this time. Jan 5th 12:07pm PST - No sign that OVH has looked at our ticket yet, and no way to contact them until business hours tomorrow. This might be a while unfortunately. Jan 6th 08:05am PST - OVH is finally investigating, hopefully back up soon. https://status.fedecan.ca/incident/466638 Sun, 24 Nov 2024 19:00:00 +0000 https://status.fedecan.ca/incident/466638#a784efea1a488595e98c2032d49aab01deef07f03825e7383297a48826e7305e Maintenance completed https://status.fedecan.ca/incident/466638 Sun, 24 Nov 2024 18:00:00 -0000 https://status.fedecan.ca/incident/466638#62b944c877e97c9940b9accb1a31674779d992e85bf18d689b71592c715bee1d Upgrading to the latest lemmy release https://status.fedecan.ca/incident/467019 Tue, 05 Nov 2024 18:30:00 +0000 https://status.fedecan.ca/incident/467019#950222bd30088e2eac769b15442fd02033e31a4b45d1a9609905b0890a4522f3 Maintenance completed https://status.fedecan.ca/incident/467019 Tue, 05 Nov 2024 16:00:00 -0000 https://status.fedecan.ca/incident/467019#7ac288c15860751d36835fa0057c2e430c526bee1c66431ea9e2c04da144217a One of our NVMe drives has failed and our service provider requires a 2 hour maintenance window to replace it. Hopefully the actual outage will be significantly shorter. https://status.fedecan.ca/incident/393710 Sat, 06 Jul 2024 19:00:00 +0000 https://status.fedecan.ca/incident/393710#f1d5fbc57ba73cf2f1327742eae7550ef41b072a1126b82a223fce3af0898dd2 Maintenance completed https://status.fedecan.ca/incident/393710 Sat, 06 Jul 2024 18:00:00 -0000 https://status.fedecan.ca/incident/393710#4018ac3d0e8eb912ca0aa730e5797164c2fa5f7cf73f85c2b3c558d40d10381c We'll be performing some tuning and upgrades to improve performance and hopefully resolve the recent instability isssues. https://status.fedecan.ca/incident/381205 Sat, 08 Jun 2024 00:30:00 +0000 https://status.fedecan.ca/incident/381205#bb8a3cde0f3f2d76600f674bbccc262eaf399e47ebc664fd2763e3bdf421b476 Maintenance completed https://status.fedecan.ca/incident/381205 Fri, 07 Jun 2024 22:50:00 -0000 https://status.fedecan.ca/incident/381205#01d9b5c2b28f12cd4c0339faeea2f4166b2ac7c6bf4906fbc0be654e941c77d9 Currently upgrading to 0.19.4, see https://lemmy.ca/post/22742132 for more details https://status.fedecan.ca/incident/327868 Sun, 18 Feb 2024 19:59:00 +0000 https://status.fedecan.ca/incident/327868#59d5a387da0f21db05f842da98668d02a3ce00972418550c2cb464c6f4293193 Maintenance completed https://status.fedecan.ca/incident/327868 Sun, 18 Feb 2024 19:00:00 -0000 https://status.fedecan.ca/incident/327868#5c8fa21fa2ce275066dffe7979d243c924c5a6c123cae6ad283d7a6b7b9b7dda Applying the latest ubuntu updates, and migrating us to use multiple lemmy / lemmy-ui containers for better resiliency. See https://lemmy.ca/post/15548660 https://status.fedecan.ca/incident/301024 Fri, 15 Dec 2023 16:44:35 +0000 https://status.fedecan.ca/incident/301024#200cb11091ef5ea756064b81257814e91ed39e8b7dd080038fe344d4ecb0afb2 Maintenance completed https://status.fedecan.ca/incident/301024 Fri, 15 Dec 2023 16:44:35 -0000 https://status.fedecan.ca/incident/301024#2a7353372e933213254def3b5a988897c227fd43ba83c7925009df8db368f5c5 Site will be down for a bit while we upgade to 0.19 https://status.fedecan.ca/incident/301024 Fri, 15 Dec 2023 16:19:00 -0000 https://status.fedecan.ca/incident/301024#2a7353372e933213254def3b5a988897c227fd43ba83c7925009df8db368f5c5 Site will be down for a bit while we upgade to 0.19 https://status.fedecan.ca/incident/266253 Sat, 30 Sep 2023 02:32:00 -0000 https://status.fedecan.ca/incident/266253#e9f92ebbcf8d8a89ac055f3e050c34df46e530ab4f0c90843dc39daa960d4527 Back online https://status.fedecan.ca/incident/266253 Sat, 30 Sep 2023 02:30:00 -0000 https://status.fedecan.ca/incident/266253#ab05b1222b544c6b7bad7234129f00169951c9a9bca070b829fbd8c8fe7df65b Tried to do a quick update to 0.18.5 but had a kernel lockup on shutting down an old container. Had to hard power cycle the server, should be back up in a few minutes. https://status.fedecan.ca/incident/257501 Sun, 10 Sep 2023 22:06:28 +0000 https://status.fedecan.ca/incident/257501#54d807ba5287853d51480d1a732b5522ec459bf3fd758492bdab1f60ed992898 Maintenance completed https://status.fedecan.ca/incident/257501 Sun, 10 Sep 2023 22:06:28 -0000 https://status.fedecan.ca/incident/257501#42a05cca8f2d1f97b521f56ff33cb3c5978a95b3c5c1bf173219c37849620d94 We're finally running low on disk space so we need to migrate our image uploads over to object storage. This will require some brief downtime at the start, then hopefully I should be able to bring the site back online while the images migrate. https://status.fedecan.ca/incident/257501 Sun, 10 Sep 2023 20:52:00 -0000 https://status.fedecan.ca/incident/257501#42a05cca8f2d1f97b521f56ff33cb3c5978a95b3c5c1bf173219c37849620d94 We're finally running low on disk space so we need to migrate our image uploads over to object storage. This will require some brief downtime at the start, then hopefully I should be able to bring the site back online while the images migrate. https://status.fedecan.ca/incident/257501 Sun, 10 Sep 2023 19:17:05 -0000 https://status.fedecan.ca/incident/257501#42a05cca8f2d1f97b521f56ff33cb3c5978a95b3c5c1bf173219c37849620d94 We're finally running low on disk space so we need to migrate our image uploads over to object storage. This will require some brief downtime at the start, then hopefully I should be able to bring the site back online while the images migrate. https://status.fedecan.ca/incident/257501 Sun, 10 Sep 2023 18:09:00 -0000 https://status.fedecan.ca/incident/257501#42a05cca8f2d1f97b521f56ff33cb3c5978a95b3c5c1bf173219c37849620d94 We're finally running low on disk space so we need to migrate our image uploads over to object storage. This will require some brief downtime at the start, then hopefully I should be able to bring the site back online while the images migrate. https://status.fedecan.ca/incident/257501 Sun, 10 Sep 2023 18:00:00 -0000 https://status.fedecan.ca/incident/257501#42a05cca8f2d1f97b521f56ff33cb3c5978a95b3c5c1bf173219c37849620d94 We're finally running low on disk space so we need to migrate our image uploads over to object storage. This will require some brief downtime at the start, then hopefully I should be able to bring the site back online while the images migrate. https://status.fedecan.ca/incident/239732 Mon, 31 Jul 2023 16:19:00 -0000 https://status.fedecan.ca/incident/239732#f5bfabc978d6c4bd597acadbea9e65b5dab3d563edeecfae8051cc1325000b9f Looking into what happened overnight that knocked us offline for a while. We lost network link on the server, so we're following up with OVH. Update: It was an OVH billing issue, even though we had auto-renew enabled they didn't renew properly :( https://status.fedecan.ca/incident/238592 Fri, 28 Jul 2023 15:14:00 -0000 https://status.fedecan.ca/incident/238592#d8873f92274c75add1d8b0c2602c65a2d40d070f3fdf61839538db7df3cd4631 08:14 - Went to upgrade to lemmy 0.18.3 and it triggered a kernel panic. System is rebooting now. 08:20 - System is back up but DB upgrades are currently running for 0.18.3, should be back online shortly. 08:30 - All sorted out and online with 0.18.3, enjoy! https://status.fedecan.ca/incident/236558 Mon, 24 Jul 2023 03:13:00 -0000 https://status.fedecan.ca/incident/236558#769a090cd46e63a805e35b1325633f4d9dc07714773c465f81ed14dd2d682aa9 Something strange is going on with the memory in our server, currently investigating but things should be back online and stable for now. Edit: Fixed fully now, was a hugepages misconfig. https://status.fedecan.ca/incident/235791 Sat, 22 Jul 2023 18:49:42 +0000 https://status.fedecan.ca/incident/235791#d4fcb7bfd1a156db915864a7d64f87bcd1abaaf4737ad4444a6c059c5e6785f0 Maintenance completed https://status.fedecan.ca/incident/235791 Sat, 22 Jul 2023 18:49:42 -0000 https://status.fedecan.ca/incident/235791#76d53e8ce640b59781d4ac5acb6c1e44dd88b7719d48eda6c8f9010ec6d9e7af We'll be taking things down for a bit to apply OS updates and do some kernel/network tuning. https://status.fedecan.ca/incident/235791 Sat, 22 Jul 2023 18:30:00 -0000 https://status.fedecan.ca/incident/235791#76d53e8ce640b59781d4ac5acb6c1e44dd88b7719d48eda6c8f9010ec6d9e7af We'll be taking things down for a bit to apply OS updates and do some kernel/network tuning. https://status.fedecan.ca/incident/235750 Fri, 21 Jul 2023 15:41:00 -0000 https://status.fedecan.ca/incident/235750#bcb2f2256f2f6aacd32d74c68ea16dca9b51e95b03afd357a3b954667cc761de Seeing more database locks again, have killed off the offending queries while we dig more. Update: This mornings slow delete query came in from the fediverse and impacted multiple lemmy sites at the same time. The core issue is https://github.com/LemmyNet/lemmy/issues/3165 https://status.fedecan.ca/incident/234523 Tue, 18 Jul 2023 20:57:00 -0000 https://status.fedecan.ca/incident/234523#5f76fc2dbaeb097b271dd18186aeccf821d7fbef250adc9df533d6ca03b43d5e Just restarted services quickly to fix the Hot feed - https://lemmy.ca/post/1685851?scrollToComments=true https://status.fedecan.ca/incident/234064 Mon, 17 Jul 2023 22:37:00 -0000 https://status.fedecan.ca/incident/234064#4ea7a35aec24903c533f0df06f5ba8202b40e58b12b5f3373af1dc001483ce2b We're hitting a lemmy bug triggered when someone deletes their account, that is locking the database. I'm disabling account removal temporarily until we can identify the fix. https://status.fedecan.ca/incident/234023 Mon, 17 Jul 2023 20:18:00 -0000 https://status.fedecan.ca/incident/234023#b686314943e25d4d989cac6902e40c56b03ae0f0e0fb5e4cc5b83f538c603e14 Saw some database locking that took things down, restarted postgres and investigating further. https://status.fedecan.ca/incident/233677 Sun, 16 Jul 2023 23:07:00 -0000 https://status.fedecan.ca/incident/233677#cae87b3913eeb39331eca853b0445ab5e76dabcde534e03738306ceff588fde6 Restarted services to do some memory tuning, we've been seeing pictrs crashing recently due to high memory consumption.