Incidents reported on status page for Fedecan https://status.fedecan.ca/ en https://status.fedecan.ca/ Sun, 12 Jul 2026 19:43:12 +0000 https://status.fedecan.ca/#4ab0fa4478c128d8f7dcdf502c372913ef819c7df1583a8ffa7d905c77ca2eae pixelfed.ca recovered https://status.fedecan.ca/ Sun, 12 Jul 2026 19:40:10 +0000 https://status.fedecan.ca/#4ab0fa4478c128d8f7dcdf502c372913ef819c7df1583a8ffa7d905c77ca2eae pixelfed.ca went down https://status.fedecan.ca/ Sat, 11 Jul 2026 07:22:15 +0000 https://status.fedecan.ca/#e08e14b0e1b837f3a5c54d6910a8bde697ca7b477e675c1eb3785809ba6be0ce piefed.ca recovered https://status.fedecan.ca/ Sat, 11 Jul 2026 07:19:08 +0000 https://status.fedecan.ca/#e08e14b0e1b837f3a5c54d6910a8bde697ca7b477e675c1eb3785809ba6be0ce piefed.ca went down https://status.fedecan.ca/ Tue, 07 Jul 2026 02:13:10 +0000 https://status.fedecan.ca/#00244b3c0cfacbacfd2ac0d360c09b4b36b3924b6877a5638381fdbf736dadbb pixelfed.ca recovered https://status.fedecan.ca/ Tue, 07 Jul 2026 02:12:56 +0000 https://status.fedecan.ca/#d0c68b4c577f06125efd9fec7737c8664811a66809f616a5235f7629f18aee4a piefed.ca recovered https://status.fedecan.ca/ Tue, 07 Jul 2026 02:10:13 +0000 https://status.fedecan.ca/#00244b3c0cfacbacfd2ac0d360c09b4b36b3924b6877a5638381fdbf736dadbb pixelfed.ca went down https://status.fedecan.ca/ Tue, 07 Jul 2026 02:10:12 +0000 https://status.fedecan.ca/#d0c68b4c577f06125efd9fec7737c8664811a66809f616a5235f7629f18aee4a piefed.ca went down https://status.fedecan.ca/ Sat, 04 Jul 2026 23:44:23 +0000 https://status.fedecan.ca/#d119637bc456c9a9c71ca64a141c52f5c715f4942a8f38a9159a22d2bad0d50a lemmy.ca recovered https://status.fedecan.ca/ Sat, 04 Jul 2026 23:43:11 +0000 https://status.fedecan.ca/#3890e3783b0b9e17cd9b6265bef5039801bbec85b35d4a25e5fffb457d9be4bc pixelfed.ca recovered https://status.fedecan.ca/ Sat, 04 Jul 2026 23:43:03 +0000 https://status.fedecan.ca/#57def2cf89560152865633dba01070b43d90698a07b8ea16a548915ca0e184c7 piefed.ca recovered https://status.fedecan.ca/ Sat, 04 Jul 2026 23:17:26 +0000 https://status.fedecan.ca/#d119637bc456c9a9c71ca64a141c52f5c715f4942a8f38a9159a22d2bad0d50a lemmy.ca went down https://status.fedecan.ca/ Sat, 04 Jul 2026 23:16:13 +0000 https://status.fedecan.ca/#3890e3783b0b9e17cd9b6265bef5039801bbec85b35d4a25e5fffb457d9be4bc pixelfed.ca went down https://status.fedecan.ca/ Sat, 04 Jul 2026 23:15:50 +0000 https://status.fedecan.ca/#57def2cf89560152865633dba01070b43d90698a07b8ea16a548915ca0e184c7 piefed.ca went down https://status.fedecan.ca/ Fri, 03 Jul 2026 16:20:26 +0000 https://status.fedecan.ca/#7c99654059cace2c1b72f4c5c39a86f06a49f7e111faf007000c1c53b1ad2b0b lemmy.ca recovered https://status.fedecan.ca/ Fri, 03 Jul 2026 16:17:25 +0000 https://status.fedecan.ca/#7c99654059cace2c1b72f4c5c39a86f06a49f7e111faf007000c1c53b1ad2b0b lemmy.ca went down https://status.fedecan.ca/ Thu, 02 Jul 2026 13:29:38 +0000 https://status.fedecan.ca/#d6ea00f8511f95e5619a3e29efafa93152467fff7d35629ba0ed03c7ce167994 lemmy.ca recovered https://status.fedecan.ca/ Thu, 02 Jul 2026 13:26:44 +0000 https://status.fedecan.ca/#d6ea00f8511f95e5619a3e29efafa93152467fff7d35629ba0ed03c7ce167994 lemmy.ca went down https://status.fedecan.ca/ Wed, 01 Jul 2026 13:56:25 +0000 https://status.fedecan.ca/#ca03682c7104cb473db1a217222479a54208b968214ba1c2e4334a1831d0378d lemmy.ca recovered https://status.fedecan.ca/ Wed, 01 Jul 2026 13:53:18 +0000 https://status.fedecan.ca/#ca03682c7104cb473db1a217222479a54208b968214ba1c2e4334a1831d0378d lemmy.ca went down https://status.fedecan.ca/ Sat, 27 Jun 2026 19:10:19 +0000 https://status.fedecan.ca/#47d19a9ca847574a2a95ee49718bc4226d046f7872e1b754be7547a0da534465 pixelfed.ca recovered https://status.fedecan.ca/ Sat, 27 Jun 2026 19:07:13 +0000 https://status.fedecan.ca/#47d19a9ca847574a2a95ee49718bc4226d046f7872e1b754be7547a0da534465 pixelfed.ca went down https://status.fedecan.ca/ Fri, 26 Jun 2026 19:19:09 +0000 https://status.fedecan.ca/#bf1736b86a15f9c498b1787dcb6ae2a29684cb677a0fbdffb1e11b1ae3e11e07 pixelfed.ca recovered https://status.fedecan.ca/ Fri, 26 Jun 2026 19:16:13 +0000 https://status.fedecan.ca/#bf1736b86a15f9c498b1787dcb6ae2a29684cb677a0fbdffb1e11b1ae3e11e07 pixelfed.ca went down https://status.fedecan.ca/ Fri, 19 Jun 2026 04:10:14 +0000 https://status.fedecan.ca/#77ecf751f62737062577ff6ae0e8ea6306c203b6833cf99a911bcad6a5944b4c pixelfed.ca recovered https://status.fedecan.ca/ Fri, 19 Jun 2026 04:09:56 +0000 https://status.fedecan.ca/#c98b0dd03b5a98ce49b91488d7431fc0ed28583130e3599e7522eca2e6bb9d6c piefed.ca recovered https://status.fedecan.ca/ Fri, 19 Jun 2026 04:08:55 +0000 https://status.fedecan.ca/#98f9b3759d05ecc7527cf225c56fae14b222933f5c191bfa82ba4e4b5c1c10a0 lemmy.ca recovered https://status.fedecan.ca/ Fri, 19 Jun 2026 04:05:04 +0000 https://status.fedecan.ca/#77ecf751f62737062577ff6ae0e8ea6306c203b6833cf99a911bcad6a5944b4c pixelfed.ca went down https://status.fedecan.ca/ Fri, 19 Jun 2026 04:04:50 +0000 https://status.fedecan.ca/#c98b0dd03b5a98ce49b91488d7431fc0ed28583130e3599e7522eca2e6bb9d6c piefed.ca went down https://status.fedecan.ca/ Fri, 19 Jun 2026 04:03:25 +0000 https://status.fedecan.ca/#98f9b3759d05ecc7527cf225c56fae14b222933f5c191bfa82ba4e4b5c1c10a0 lemmy.ca went down https://status.fedecan.ca/ Fri, 19 Jun 2026 03:56:21 +0000 https://status.fedecan.ca/#b3992e243983796b814cc076dd3a7bc47859da604080a01fff67fad08318850d lemmy.ca recovered https://status.fedecan.ca/ Fri, 19 Jun 2026 03:55:15 +0000 https://status.fedecan.ca/#62018bf7673994be08763f7d5086ef62852f45033fa9bc60514bd984376022a4 pixelfed.ca recovered https://status.fedecan.ca/ Fri, 19 Jun 2026 03:55:10 +0000 https://status.fedecan.ca/#f564cc5567e08f8a8cc94f6a4cd55c89ec5fff05d263c745a6acb9b781d46b4b piefed.ca recovered https://status.fedecan.ca/ Fri, 19 Jun 2026 03:54:19 +0000 https://status.fedecan.ca/#b3992e243983796b814cc076dd3a7bc47859da604080a01fff67fad08318850d lemmy.ca went down https://status.fedecan.ca/ Fri, 19 Jun 2026 03:53:05 +0000 https://status.fedecan.ca/#62018bf7673994be08763f7d5086ef62852f45033fa9bc60514bd984376022a4 pixelfed.ca went down https://status.fedecan.ca/ Fri, 19 Jun 2026 03:52:45 +0000 https://status.fedecan.ca/#f564cc5567e08f8a8cc94f6a4cd55c89ec5fff05d263c745a6acb9b781d46b4b piefed.ca went down https://status.fedecan.ca/ Fri, 19 Jun 2026 03:39:54 +0000 https://status.fedecan.ca/#bc494780a6b9ba40705804f27d1439136b431e4338a1637b5c378e320df211a0 piefed.ca recovered https://status.fedecan.ca/ Fri, 19 Jun 2026 03:37:01 +0000 https://status.fedecan.ca/#bc494780a6b9ba40705804f27d1439136b431e4338a1637b5c378e320df211a0 piefed.ca went down https://status.fedecan.ca/incident/928370 Fri, 19 Jun 2026 03:30:00 -0000 https://status.fedecan.ca/incident/928370#666106c34b6af109bbc67a6b6bc0887dc08f7423b91671eb57cf6fcfad9a078a All services will be unavailable while we apply OS patches https://status.fedecan.ca/ Wed, 10 Jun 2026 13:04:06 +0000 https://status.fedecan.ca/#e3b36fcdc126f230c642695913dfc6057ff9827f9a4157ad702970d64e92ec8c piefed.ca recovered https://status.fedecan.ca/ Wed, 10 Jun 2026 13:01:03 +0000 https://status.fedecan.ca/#e3b36fcdc126f230c642695913dfc6057ff9827f9a4157ad702970d64e92ec8c piefed.ca went down https://status.fedecan.ca/ Wed, 10 Jun 2026 12:28:03 +0000 https://status.fedecan.ca/#343ef697aa8d940da10f16cb216b1974899817b7ba27257aeba3641c1212e159 piefed.ca recovered https://status.fedecan.ca/ Wed, 10 Jun 2026 12:25:16 +0000 https://status.fedecan.ca/#343ef697aa8d940da10f16cb216b1974899817b7ba27257aeba3641c1212e159 piefed.ca went down https://status.fedecan.ca/ Wed, 10 Jun 2026 12:16:20 +0000 https://status.fedecan.ca/#12a369d173454123bb9e3c9b428a5517e82f6f3c2d956b243ff64c9bbf536726 piefed.ca recovered https://status.fedecan.ca/ Wed, 10 Jun 2026 12:10:11 +0000 https://status.fedecan.ca/#12a369d173454123bb9e3c9b428a5517e82f6f3c2d956b243ff64c9bbf536726 piefed.ca went down https://status.fedecan.ca/ Tue, 09 Jun 2026 23:02:30 +0000 https://status.fedecan.ca/#f00a2cdfde9d48bc2f539f1fcacb9456663ad00827dd1930298e6fe3e80951cc lemmy.ca recovered https://status.fedecan.ca/ Tue, 09 Jun 2026 22:59:26 +0000 https://status.fedecan.ca/#f00a2cdfde9d48bc2f539f1fcacb9456663ad00827dd1930298e6fe3e80951cc lemmy.ca went down https://status.fedecan.ca/ Wed, 03 Jun 2026 19:05:24 +0000 https://status.fedecan.ca/#9573868b2df6666a60f15198b8e990bf3de10a548ea1393acb919ed0f2fd7620 lemmy.ca recovered https://status.fedecan.ca/ Wed, 03 Jun 2026 19:02:29 +0000 https://status.fedecan.ca/#9573868b2df6666a60f15198b8e990bf3de10a548ea1393acb919ed0f2fd7620 lemmy.ca went down https://status.fedecan.ca/ Wed, 03 Jun 2026 14:18:55 +0000 https://status.fedecan.ca/#ab90088070460337c2f674141d1cfd3d257a81a416162965f7fd73a23e48e65e piefed.ca recovered https://status.fedecan.ca/ Wed, 03 Jun 2026 13:49:02 +0000 https://status.fedecan.ca/#ab90088070460337c2f674141d1cfd3d257a81a416162965f7fd73a23e48e65e piefed.ca went down https://status.fedecan.ca/ Sun, 31 May 2026 01:56:24 +0000 https://status.fedecan.ca/#f372900bfd4c98d4168302dfdfb4cacab95bb4adf58eb6426c5a6e1c98f89a92 lemmy.ca recovered https://status.fedecan.ca/ Sun, 31 May 2026 01:53:27 +0000 https://status.fedecan.ca/#f372900bfd4c98d4168302dfdfb4cacab95bb4adf58eb6426c5a6e1c98f89a92 lemmy.ca went down https://status.fedecan.ca/ Sat, 30 May 2026 23:29:22 +0000 https://status.fedecan.ca/#5ca58926fdf005f23f56abe4c830f638c5eb53ab4753f43593026574c05da1a8 lemmy.ca recovered https://status.fedecan.ca/ Sat, 30 May 2026 23:26:54 +0000 https://status.fedecan.ca/#5ca58926fdf005f23f56abe4c830f638c5eb53ab4753f43593026574c05da1a8 lemmy.ca went down https://status.fedecan.ca/ Tue, 26 May 2026 09:47:19 +0000 https://status.fedecan.ca/#e7320fc73e7a4cf18ed54d7d237001f905860b4b40f61470d5adc3340587aeed lemmy.ca recovered https://status.fedecan.ca/ Tue, 26 May 2026 09:44:33 +0000 https://status.fedecan.ca/#e7320fc73e7a4cf18ed54d7d237001f905860b4b40f61470d5adc3340587aeed lemmy.ca went down https://status.fedecan.ca/ Mon, 25 May 2026 21:58:00 +0000 https://status.fedecan.ca/#d407be5b1734ebd23a7126ba68410250d7541ac6c465ba23949229924d0d444b piefed.ca recovered https://status.fedecan.ca/ Mon, 25 May 2026 21:54:57 +0000 https://status.fedecan.ca/#d407be5b1734ebd23a7126ba68410250d7541ac6c465ba23949229924d0d444b piefed.ca went down https://status.fedecan.ca/ Mon, 18 May 2026 17:21:54 +0000 https://status.fedecan.ca/#d09119cac4ed73e3327a2168fb3fdd4db484dafdd500459f366d46eea220ff2a piefed.ca recovered https://status.fedecan.ca/ Mon, 18 May 2026 17:18:56 +0000 https://status.fedecan.ca/#d09119cac4ed73e3327a2168fb3fdd4db484dafdd500459f366d46eea220ff2a piefed.ca went down https://status.fedecan.ca/ Mon, 18 May 2026 17:02:22 +0000 https://status.fedecan.ca/#ac4deca8ebef1f018216dda87959ae424ed77e1ebd60f396966b84e2ff8e37c7 lemmy.ca recovered https://status.fedecan.ca/ Mon, 18 May 2026 16:59:20 +0000 https://status.fedecan.ca/#ac4deca8ebef1f018216dda87959ae424ed77e1ebd60f396966b84e2ff8e37c7 lemmy.ca went down https://status.fedecan.ca/ Sun, 17 May 2026 17:07:12 +0000 https://status.fedecan.ca/#b5681ed5c0918f4ee85681e8f51d51c76a7c8839bbb4231a516d21da977c8cf0 pixelfed.ca recovered https://status.fedecan.ca/ Sun, 17 May 2026 17:04:37 +0000 https://status.fedecan.ca/#b5681ed5c0918f4ee85681e8f51d51c76a7c8839bbb4231a516d21da977c8cf0 pixelfed.ca went down https://status.fedecan.ca/ Sat, 16 May 2026 20:19:11 +0000 https://status.fedecan.ca/#27868351c9bd75718997f1c0b0e9c6b6ad1506e06829854c9ba725218edfa4fe pixelfed.ca recovered https://status.fedecan.ca/ Sat, 16 May 2026 20:18:51 +0000 https://status.fedecan.ca/#2cff07cd1f0a593a551b830a0d498d4b866205ef3eb1a16e5b8143794afe830d piefed.ca recovered https://status.fedecan.ca/ Sat, 16 May 2026 20:16:18 +0000 https://status.fedecan.ca/#27868351c9bd75718997f1c0b0e9c6b6ad1506e06829854c9ba725218edfa4fe pixelfed.ca went down https://status.fedecan.ca/ Sat, 16 May 2026 20:16:17 +0000 https://status.fedecan.ca/#2cff07cd1f0a593a551b830a0d498d4b866205ef3eb1a16e5b8143794afe830d piefed.ca went down https://status.fedecan.ca/ Tue, 12 May 2026 18:35:28 +0000 https://status.fedecan.ca/#3fc1567cd87bb0c88b33b65eac1d341d7eb82085896d87c2f0b228646ef57b2d lemmy.ca recovered https://status.fedecan.ca/ Tue, 12 May 2026 18:32:28 +0000 https://status.fedecan.ca/#3fc1567cd87bb0c88b33b65eac1d341d7eb82085896d87c2f0b228646ef57b2d lemmy.ca went down https://status.fedecan.ca/incident/855012 Sun, 22 Mar 2026 21:04:50 -0000 https://status.fedecan.ca/incident/855012#a1e4e6c1ded9b601cb383cf89735edb909e67d27656a866f0fd03a07564a0b96 We're moving away from Cloudflare. No impact is expected, but you may see some brief errors as we update configs. https://status.fedecan.ca/incident/826639 Mon, 16 Feb 2026 16:25:00 -0000 https://status.fedecan.ca/incident/826639#95c6e838a98827f6e26908ab73c7412148729321f31a991f96e42f3518e999d6 Pixelfed.ca is currently being migrated off Cloudflare. https://status.fedecan.ca/incident/816460 Sun, 01 Feb 2026 22:58:00 -0000 https://status.fedecan.ca/incident/816460#7741b77b62860b7c3ce480d90865fc55ba41110a6295ababfc75495573b78006 Unfortunately upgrading to the latest piefed revealed an encoding issue with our database, that requires a dump + restore of the database. This should take under an hour, expect us back online before 4pm PST. https://status.fedecan.ca/incident/816438 Sun, 01 Feb 2026 21:25:00 -0000 https://status.fedecan.ca/incident/816438#5cfb234667e1d48e03b82641c3713f91eae303043214b9bec7810c136193b28f Upgrading to the latest piefed, back in a few mins! https://status.fedecan.ca/incident/793369 Wed, 24 Dec 2025 04:00:00 -0000 https://status.fedecan.ca/incident/793369#441279275cfed818f34cc4cc97030dceee49a2dcbc81ca777b1396cd065279fb Upgrading us to the latest piefed 1.4 https://status.fedecan.ca/incident/541995 Tue, 08 Apr 2025 16:30:09 -0000 https://status.fedecan.ca/incident/541995#509305b82f70c551b6804a38bb5cbb95048ad0953788081099f0f0b671327230 Upgrading to 0.19.11 https://status.fedecan.ca/incident/502669 Thu, 30 Jan 2025 18:00:43 -0000 https://status.fedecan.ca/incident/502669#2662c3f20e877017c24e24c9410a5ec4b37cdb9bf3a1a0d20b40f7cb1f5b5334 We'll be offline for about 2 hours as we migrate pictrs from it's internal sled database, over to our postgres deployment https://status.fedecan.ca/incident/502668 Wed, 29 Jan 2025 18:00:58 -0000 https://status.fedecan.ca/incident/502668#c927c18c0c838323b9ef0fa5333ab14b5f79dd59e73d466bd952e9c8803b088b We'll be offline while we migrate from OVH to our new dedicated hardware. https://status.fedecan.ca/incident/492702 Wed, 08 Jan 2025 18:00:34 -0000 https://status.fedecan.ca/incident/492702#e8ea47190cb6725d6b2931dc29fa625ed208b6d153518cc1eda1cadc476b59fe Taking the site down for about 5 minutes, so that we can get a consistent copy of our databases (postgres + pict-rs sled). https://status.fedecan.ca/incident/492083 Tue, 07 Jan 2025 16:42:00 -0000 https://status.fedecan.ca/incident/492083#31c0260d5d385312faeec86b3b5cfea4c1cf2229beec7737c131d39a87fe988a We're back again.... https://status.fedecan.ca/incident/492083 Tue, 07 Jan 2025 16:36:00 -0000 https://status.fedecan.ca/incident/492083#37572d7a6db7a7bc4f40e8abd7b8e4412ab1fa1e7e5e6fb071fa4bd4ca4856df They seem to be replacing our power supply again https://status.fedecan.ca/incident/492083 Tue, 07 Jan 2025 16:11:00 -0000 https://status.fedecan.ca/incident/492083#21fa02250e03b9f526535b689e357646c88dfcaadc6c1f2c1af56fa76a05deaa Ahhhh the server is down again. Talking to OVH to get us back up ASAP. https://status.fedecan.ca/incident/491086 Mon, 06 Jan 2025 20:21:00 -0000 https://status.fedecan.ca/incident/491086#a96b5510f6d9228eea378143b3926ef84af0199b1b138e9707ad1ba247dc20d4 We're back! https://lemmy.ca/post/36540245 https://status.fedecan.ca/incident/491086 Mon, 06 Jan 2025 19:18:00 -0000 https://status.fedecan.ca/incident/491086#c720b921cfd70034c56fcc2a7e80b703e26245e8fd8199878ee76adb0038e034 We're still waiting on OVH to repair the hardware, the datacenter tech has been working on it for about 3 hours now. Hopefully back soon! https://status.fedecan.ca/incident/491086 Sun, 05 Jan 2025 18:19:00 -0000 https://status.fedecan.ca/incident/491086#cae41b10df14806c503c9c5990f7580d72d27ccbced0a234ba6e45eacc8a2a7c Jan 5th 10:19am PST - Our server appears to have had a physical hardware failure. We have a support ticket open with OVH, but no ETA on repairs at this time. Jan 5th 12:07pm PST - No sign that OVH has looked at our ticket yet, and no way to contact them until business hours tomorrow. This might be a while unfortunately. Jan 6th 08:05am PST - OVH is finally investigating, hopefully back up soon. https://status.fedecan.ca/incident/466638 Sun, 24 Nov 2024 18:00:00 -0000 https://status.fedecan.ca/incident/466638#62b944c877e97c9940b9accb1a31674779d992e85bf18d689b71592c715bee1d Upgrading to the latest lemmy release https://status.fedecan.ca/incident/467019 Tue, 05 Nov 2024 16:00:00 -0000 https://status.fedecan.ca/incident/467019#7ac288c15860751d36835fa0057c2e430c526bee1c66431ea9e2c04da144217a One of our NVMe drives has failed and our service provider requires a 2 hour maintenance window to replace it. Hopefully the actual outage will be significantly shorter. https://status.fedecan.ca/incident/393710 Sat, 06 Jul 2024 18:00:00 -0000 https://status.fedecan.ca/incident/393710#4018ac3d0e8eb912ca0aa730e5797164c2fa5f7cf73f85c2b3c558d40d10381c We'll be performing some tuning and upgrades to improve performance and hopefully resolve the recent instability isssues. https://status.fedecan.ca/incident/381205 Fri, 07 Jun 2024 22:50:00 -0000 https://status.fedecan.ca/incident/381205#01d9b5c2b28f12cd4c0339faeea2f4166b2ac7c6bf4906fbc0be654e941c77d9 Currently upgrading to 0.19.4, see https://lemmy.ca/post/22742132 for more details https://status.fedecan.ca/incident/327868 Sun, 18 Feb 2024 19:00:00 -0000 https://status.fedecan.ca/incident/327868#5c8fa21fa2ce275066dffe7979d243c924c5a6c123cae6ad283d7a6b7b9b7dda Applying the latest ubuntu updates, and migrating us to use multiple lemmy / lemmy-ui containers for better resiliency. See https://lemmy.ca/post/15548660 https://status.fedecan.ca/incident/301024 Fri, 15 Dec 2023 16:19:00 -0000 https://status.fedecan.ca/incident/301024#2a7353372e933213254def3b5a988897c227fd43ba83c7925009df8db368f5c5 Site will be down for a bit while we upgade to 0.19 https://status.fedecan.ca/incident/266253 Sat, 30 Sep 2023 02:32:00 -0000 https://status.fedecan.ca/incident/266253#e9f92ebbcf8d8a89ac055f3e050c34df46e530ab4f0c90843dc39daa960d4527 Back online https://status.fedecan.ca/incident/266253 Sat, 30 Sep 2023 02:30:00 -0000 https://status.fedecan.ca/incident/266253#ab05b1222b544c6b7bad7234129f00169951c9a9bca070b829fbd8c8fe7df65b Tried to do a quick update to 0.18.5 but had a kernel lockup on shutting down an old container. Had to hard power cycle the server, should be back up in a few minutes. https://status.fedecan.ca/incident/257501 Sun, 10 Sep 2023 18:00:00 -0000 https://status.fedecan.ca/incident/257501#42a05cca8f2d1f97b521f56ff33cb3c5978a95b3c5c1bf173219c37849620d94 We're finally running low on disk space so we need to migrate our image uploads over to object storage. This will require some brief downtime at the start, then hopefully I should be able to bring the site back online while the images migrate. https://status.fedecan.ca/incident/239732 Mon, 31 Jul 2023 16:19:00 -0000 https://status.fedecan.ca/incident/239732#f5bfabc978d6c4bd597acadbea9e65b5dab3d563edeecfae8051cc1325000b9f Looking into what happened overnight that knocked us offline for a while. We lost network link on the server, so we're following up with OVH. Update: It was an OVH billing issue, even though we had auto-renew enabled they didn't renew properly :( https://status.fedecan.ca/incident/238592 Fri, 28 Jul 2023 15:14:00 -0000 https://status.fedecan.ca/incident/238592#d8873f92274c75add1d8b0c2602c65a2d40d070f3fdf61839538db7df3cd4631 08:14 - Went to upgrade to lemmy 0.18.3 and it triggered a kernel panic. System is rebooting now. 08:20 - System is back up but DB upgrades are currently running for 0.18.3, should be back online shortly. 08:30 - All sorted out and online with 0.18.3, enjoy! https://status.fedecan.ca/incident/236558 Mon, 24 Jul 2023 03:13:00 -0000 https://status.fedecan.ca/incident/236558#769a090cd46e63a805e35b1325633f4d9dc07714773c465f81ed14dd2d682aa9 Something strange is going on with the memory in our server, currently investigating but things should be back online and stable for now. Edit: Fixed fully now, was a hugepages misconfig. https://status.fedecan.ca/incident/235791 Sat, 22 Jul 2023 18:49:42 -0000 https://status.fedecan.ca/incident/235791#d4fcb7bfd1a156db915864a7d64f87bcd1abaaf4737ad4444a6c059c5e6785f0 Maintenance completed https://status.fedecan.ca/incident/235791 Sat, 22 Jul 2023 18:30:00 -0000 https://status.fedecan.ca/incident/235791#76d53e8ce640b59781d4ac5acb6c1e44dd88b7719d48eda6c8f9010ec6d9e7af We'll be taking things down for a bit to apply OS updates and do some kernel/network tuning. https://status.fedecan.ca/incident/235750 Fri, 21 Jul 2023 15:41:00 -0000 https://status.fedecan.ca/incident/235750#bcb2f2256f2f6aacd32d74c68ea16dca9b51e95b03afd357a3b954667cc761de Seeing more database locks again, have killed off the offending queries while we dig more. Update: This mornings slow delete query came in from the fediverse and impacted multiple lemmy sites at the same time. The core issue is https://github.com/LemmyNet/lemmy/issues/3165 https://status.fedecan.ca/incident/234523 Tue, 18 Jul 2023 20:57:00 -0000 https://status.fedecan.ca/incident/234523#5f76fc2dbaeb097b271dd18186aeccf821d7fbef250adc9df533d6ca03b43d5e Just restarted services quickly to fix the Hot feed - https://lemmy.ca/post/1685851?scrollToComments=true https://status.fedecan.ca/incident/234064 Mon, 17 Jul 2023 22:37:00 -0000 https://status.fedecan.ca/incident/234064#4ea7a35aec24903c533f0df06f5ba8202b40e58b12b5f3373af1dc001483ce2b We're hitting a lemmy bug triggered when someone deletes their account, that is locking the database. I'm disabling account removal temporarily until we can identify the fix. https://status.fedecan.ca/incident/234023 Mon, 17 Jul 2023 20:18:00 -0000 https://status.fedecan.ca/incident/234023#b686314943e25d4d989cac6902e40c56b03ae0f0e0fb5e4cc5b83f538c603e14 Saw some database locking that took things down, restarted postgres and investigating further. https://status.fedecan.ca/incident/233677 Sun, 16 Jul 2023 23:07:00 -0000 https://status.fedecan.ca/incident/233677#cae87b3913eeb39331eca853b0445ab5e76dabcde534e03738306ceff588fde6 Restarted services to do some memory tuning, we've been seeing pictrs crashing recently due to high memory consumption.