19:36 <alatiera> have been getting 502s and 504s for a bit now

19:36 <alatiera> known issue?

19:38 <ocrete> I'm seeing the same thing..

19:48 <airlied> yeah gl is hiccuping

19:56 <alextee> I'm getting 502 when cloning, is something wrong with infra?

19:56 <airlied> yes gitlab is having issues

20:21 <mupuf> bentiss, daniels: ^

20:21 <mupuf> In case you are still up

20:24 <craftyguy> ahh just came here to ring the bell, but I'm late

20:26 <alyssa> craftyguy: party's just getting started 🎉

20:26 <craftyguy> heh

20:47 <alyssa> a classic

21:02 <emersion> ♥

21:20 <keithp> thanks daniels!

21:43 <ngcortes> there a scheduled downtime on gitlab?

21:49 <emersion> nope, outage, being worked on

21:50 <ngcortes> word, tnx

21:51 <alyssa> ngcortes: daniels scheduled it at 21:02 UTC of course ;)

21:52 <ngcortes> alyssa, guess it's lunchtime :)

22:22 <daniels> bentiss: I have no idea what's going on with ceph. osd-26 is failing repeatedly with:

22:22 <daniels> debug 2021-12-14T22:17:38.270+0000 7f864936e080 -1 bluefs _allocate allocation failed, needed 0x807e9

22:22 <daniels> debug 2021-12-14T22:17:38.270+0000 7f864936e080 1 bluefs _allocate unable to allocate 0x90000 on bdev 1, allocator name block, allocator type hybrid, capacity 0x6fc8400000, block size 0x1000, free 0x8e21d5000, fragmentation 0.348327, allocated 0x0

22:22 <daniels> debug 2021-12-14T22:17:38.270+0000 7f864936e080 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x807e9

22:22 <daniels> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f864936e080 time 2021-12-14T22:17:38.274652+0000

22:22 <daniels> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/os/bluestore/BlueFS.cc: 2729: ceph_abort_msg("bluefs enospc")

22:23 <daniels> so like ... why can't it allocate 589824 (bytes? blocks?) when it has 38153310208 free

22:27 <bentiss> daniels: sorry I'm burned out today

22:28 <bentiss> daniels: have you cordoned server-4?

22:30 <craftyguy> is the free space there referring to disk or RAM?

22:33 <bentiss> daniels: I restarted k3s on server-4 after uncordoning it, the osd in pending are now up

22:33 <bentiss> craftyguy: no idea :(

22:44 <bentiss> daniels: I also kicked osd-0, it went in a bad state, which means we have only one osd down (the 26) with the error above

22:47 <craftyguy> if it's trying to create a file (I have no idea how bluefs works), then maybe the fs is out of free inodes?

22:47 <bentiss> maybe it's something like that

22:47 <bentiss> it seems everything is back up, but we probably need to audit tomorrow why the disks are almost full

22:48 <daniels> bentiss: oh I'm sorry to hear :( and yeah, from what I can see it's always just been 26 which is failing, but it's not clear why since the underlying storage seems like it has enough space - but I'm also frantically trying to learn how ceph actually works so I can see what that underlying storage is

22:48 <daniels> craftyguy: oh hmm

22:49 <bentiss> daniels: ceph df gives

22:49 <bentiss> POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL

22:49 <bentiss> replicapool-ssd 4 32 715 GiB 188.57k 2.1 TiB 75.70 230 GiB

22:49 <bentiss> and 75% is close enough to the limits where the disks are stopping

22:50 <bentiss> so we have only 230GB of free space, but I think we have a watermark ar 90%

22:50 <craftyguy> what does df -i show?

22:50 <craftyguy> that should show inode usage

22:52 <bentiss> craftyguy: the ceph disks don't appear on the host 'df'

22:54 <bentiss> anyway, it seems to be OK for the past 5 min, going to bed

22:55 <craftyguy> thanks for getting it running again! back to work for me, break's over

22:56 <bentiss> sorry :)

22:59 <daniels> thanks very much indeed

22:59 <daniels> sleep well, take care

23:10 <daniels> heh, I wonder if this explains it:

23:10 <daniels> pvc-db29ccdf-2e89-4acc-88ee-48fe68c13bd9 520Gi RWO Retain Released old-cluster/repo-data-gitlab-prod-gitaly-no-replicas-0 rook-ceph-block-ssd 189d