06:37 <bentiss> karolherbst: AFAICT, the fork is handled by https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/gitaly/service/repository/fork.go and it only does the clone

06:37 <bentiss> the gc should be run in background

06:44 <bentiss> karolherbst: actually... the code path from gitlab is doing slightly more than just the gitaly call: app/workers/repository_fork_worker.rb calls gitaly_fork, link_lfs_objects (which should not take any time on drm/nouveau), then project.after_import

06:46 <bentiss> project.after_import is defined in app/models/project.rb and it does a few things *before* marking the reporsitory as imported (but the thing that should take long is DetectRepositoryLanguagesWorker.perform_async, which is called asynchronously)

06:58 <bentiss> FWIW, I am waiting for the backup to finish cloning the repos, and I'll upgrade rook (the ceph operator) to 1.6.8

06:58 <bentiss> should happen in ~1h

08:36 <bentiss> OK, packing up tar, we can start the rook upgrade

09:01 <bentiss> hmm. rook upgrade finished, but seems that gitlab is not happy

09:13 <bentiss> I tried kicking postgres, but it doesn't come back up

09:24 <bentiss> daniels: I think I must be hitting https://github.com/rook/rook/issues/8085

09:33 <bentiss> daniels: enabling enableCSIHostNetwork on rook, but refreshing the cluster is going to take a little bit of time

09:50 <bentiss> FWIW, rebooting every node in the cluster one by one to get the services back

10:04 <karolherbst> bentiss: is there a way to see through logging what is taking so long?

10:04 <karolherbst> would be interesting to see

10:05 <bentiss> karolherbst: maybe, but right now I have some things to do...

10:05 <karolherbst> no worries

10:06 <bentiss> OK, indico is back

10:44 <bentiss> daniels: I managed to reboot every serrvers bu large-7. This one has the backup still uploading, so I'd rather not interrupt it

10:45 <bentiss> I am currently copying locally this morning backup into /var/openebs/local/ in case we get something wrong

10:46 <bentiss> daniels: AFAICT, gitlab is happy, but elastic-search/kibana is not

11:44 <daniels> bentiss: if it's any consolation, I spent the morning trying to get my laptop to turn on ...

11:44 <daniels> fixed now at least

12:11 <bentiss> daniels: heh, ok :)

12:11 <bentiss> daniels: so /var/openebs/local/1628308809_2021_08_07_14.1.1_gitlab_backup.tar is the last backup before I upgraded rook

12:12 <bentiss> backup upload is now completed, rebooting large-7 to enable elasticsearch

12:15 <bentiss> daniels: also, I think I'll conduct the ceph upgrade next week, the time for people to see if there was any errors during today's outage

12:19 <daniels> is the Ceph upgrade difficult/destructive, or?

12:20 <bentiss> daniels: should be all hanlded by the rook operator

12:20 <bentiss> it takes the OSD down and upgrade them one by one

12:20 <bentiss> it's almost without people noticing except when the S3 server gets restarted

12:21 <bentiss> but I'd rather leave the cluster in a working state a bit before doing the upgrade

12:43 <daniels> awesome

12:45 <bentiss> alright, kibana is now up, I am not touching the cluster anymore