Failed upgrade from 1.20.5 -> 1.21.0

I tried to upgrade my instance from 1.20.5 → 1.21.0 and am getting the following:

2023/11/17 13:24:16 cmd/web.go:194:serveInstalled() [I] PING DATABASE sqlite3
2023/11/17 13:24:16 ...ations/migrations.go:641:Migrate() [I] Migration[263]: Add git_size and lfs_size columns to repository table
2023/11/17 13:24:16 routers/common/db.go:36:InitDBEngine() [E] ORM engine initialization attempt #1/10 failed. Error: migrate: migration[263]: Add git_size and lfs_size columns to repository table failed: NOT NULL constraint failed: repository.git_size
2023/11/17 13:24:16 routers/common/db.go:37:InitDBEngine() [I] Backing off for 3 seconds
2023/11/17 13:24:19 routers/common/db.go:30:InitDBEngine() [I] ORM engine initialization attempt #2/10...
2023/11/17 13:24:19 cmd/web.go:194:serveInstalled() [I] PING DATABASE sqlite3
2023/11/17 13:24:19 ...ations/migrations.go:641:Migrate() [I] Migration[263]: Add git_size and lfs_size columns to repository table
2023/11/17 13:24:19 routers/common/db.go:36:InitDBEngine() [E] ORM engine initialization attempt #2/10 failed. Error: migrate: migration[263]: Add git_size and lfs_size columns to repository table failed: NOT NULL constraint failed: repository.git_size
2023/11/17 13:24:19 routers/common/db.go:37:InitDBEngine() [I] Backing off for 3 seconds
2023/11/17 13:24:22 routers/common/db.go:30:InitDBEngine() [I] ORM engine initialization attempt #3/10...
2023/11/17 13:24:22 cmd/web.go:194:serveInstalled() [I] PING DATABASE sqlite3
2023/11/17 13:24:22 ...ations/migrations.go:641:Migrate() [I] Migration[263]: Add git_size and lfs_size columns to repository table
2023/11/17 13:24:22 routers/common/db.go:36:InitDBEngine() [E] ORM engine initialization attempt #3/10 failed. Error: migrate: migration[263]: Add git_size and lfs_size columns to repository table failed: NOT NULL constraint failed: repository.git_size
2023/11/17 13:24:22 routers/common/db.go:37:InitDBEngine() [I] Backing off for 3 seconds
2023/11/17 13:24:25 routers/common/db.go:30:InitDBEngine() [I] ORM engine initialization attempt #4/10...
2023/11/17 13:24:25 cmd/web.go:194:serveInstalled() [I] PING DATABASE sqlite3
2023/11/17 13:24:25 ...ations/migrations.go:641:Migrate() [I] Migration[263]: Add git_size and lfs_size columns to repository table
2023/11/17 13:24:25 routers/common/db.go:36:InitDBEngine() [E] ORM engine initialization attempt #4/10 failed. Error: migrate: migration[263]: Add git_size and lfs_size columns to repository table failed: NOT NULL constraint failed: repository.git_size
2023/11/17 13:24:25 routers/common/db.go:37:InitDBEngine() [I] Backing off for 3 seconds
2023/11/17 13:24:26 ...eful/manager_unix.go:198:handleSignals() [W] PID 12297. Received SIGTERM. Shutting down...
2023/11/17 13:24:26 .../graceful/manager.go:168:doHammerTime() [W] Setting Hammer condition
2023/11/17 13:24:27 .../graceful/manager.go:184:doTerminate() [W] Terminating
2023/11/17 13:24:27 ...eful/manager_unix.go:208:handleSignals() [W] PID: 12297. Background context for manager closed - context canceled - Shutting down...
2023/11/17 13:24:28 routers/init.go:69:mustInitCtx() [F] code.gitea.io/gitea/routers/common.InitDBEngine(ctx) failed: Aborted due to shutdown:
in retry ORM engine initialization

Any thoughts on this issue?

Does 1.20.5 still run?
If so my advice would be to use gitea doctor (1.20.5 binary) on thee setup and allow it to repair aspects of the db as well as structure

gitea doctor --all --fix

I have gotten into the habit of always running this prior to upgrade “just in case”

Yes, I reverted back to 1.20.5, restored a snapshot of the db before upgrade, and it is running fine. Thanks for the doctor idea – will try that.

I ran gitea doctor --all --fix with v1.20.5 and there is one error:

[10] Check consistency of database                      
 - [E] Error: no such table: branch whilst counting Branches without existing repository                                                     
ERROR                  

Then I tried to run 1.21.1:

2023/12/11 16:25:13 ...ations/migrations.go:641:Migrate() [I] Migration[260]: Drop custom_labels column of action_runner table
2023/12/11 16:25:13 ...ations/migrations.go:641:Migrate() [I] Migration[261]: Add variable table
2023/12/11 16:25:13 ...ations/migrations.go:641:Migrate() [I] Migration[262]: Add TriggerEvent to action_run table
2023/12/11 16:25:13 ...ations/migrations.go:641:Migrate() [I] Migration[263]: Add git_size and lfs_size columns to repository table
2023/12/11 16:25:13 routers/common/db.go:36:InitDBEngine() [E] ORM engine initialization attempt #1/10 failed. Error: migrate: migration[263]: Add git_size and lfs_size columns to repository table failed: NOT NULL constraint failed: repository.git_size
2023/12/11 16:25:13 routers/common/db.go:37:InitDBEngine() [I] Backing off for 3 seconds
2023/12/11 16:25:16 routers/common/db.go:30:InitDBEngine() [I] ORM engine initialization attempt #2/10...
2023/12/11 16:25:16 cmd/web.go:194:serveInstalled() [I] PING DATABASE sqlite3
2023/12/11 16:25:16 ...ations/migrations.go:641:Migrate() [I] Migration[263]: Add git_size and lfs_size columns to repository table
2023/12/11 16:25:16 routers/common/db.go:36:InitDBEngine() [E] ORM engine initialization attempt #2/10 failed. Error: migrate: migration[263]: Add git_size and lfs_size columns to repository table failed: NOT NULL constraint failed: repository.git_size
2023/12/11 16:25:16 routers/common/db.go:37:InitDBEngine() [I] Backing off for 3 seconds

I can’t run gitea doctor with 1.21 as the DB has not been migrated yet.

So I then ran a dump and restored the db back into sqlite db and got the following during sqlite3 restore:

... 
Error: near line 31356: NOT NULL constraint failed: repository.size
Error: near line 31358: NOT NULL constraint failed: repository.size
Error: near line 31359: NOT NULL constraint failed: repository.size
Error: near line 36256: Expression tree is too large (maximum depth 1000)
... (lots of above)
Error: near line 36257: Expression tree is too large (maximum depth 1000)

(the sqlite3 restore took a long time …)

1.21.1 one now starts, but when I try to log in, I get:

Failed authentication attempt for … from … user does not exist

I’ve been upgrading a Gitea release pretty much every release for years with no problems. This is the first problem I’ve had …

It is interesting that Forgejo upgraded my instance just fine …

This was fixed by handle repository.size column being NULL in migration v263 by thenaterhood · Pull Request #28336 · go-gitea/gitea · GitHub
The reason it is fixed in Forgejo is because they have a released version with the fix in it and we are currently waiting on a few backports.
1.21.2 changelog by techknowlogick · Pull Request #28387 · go-gitea/gitea · GitHub

1 Like

Gitea upgrade to 1.21.2 worked – thanks!

1 Like