Unable to ssh to inbuilt ssh server port 2222

I have installed Gitea on Ubuntu server 20.10.
I changed the port it runs on to 3006 to avoid a conflict with Mastodon that runs on the same port.
I have changed the settings so that it should be using port 2222 for ssh.

[server]

START_SSH_SERVER = true
LOCAL_ROOT_URL = http://localhost:3006/
SSH_DOMAIN = localhost
DOMAIN = localhost
HTTP_PORT = 3006
ROOT_URL = http://git.sedulust.com/
DISABLE_SSH = false
#SSH_PORT = 22
LFS_START_SERVER = true
LFS_CONTENT_PATH = /var/lib/gitea/data/lfs
LFS_JWT_SECRET =
OFFLINE_MODE = false
SSH_PORT = 2222
[log]
MODE = console
LEVEL = warn
ROOT_PATH = /var/lib/gitea/log
REDIRECT_MACARON_LOG = true
MACARON = console
ROUTER = console

I have enabled this port in ufw.

I can’t ssh to it an nmap gives:

PORT     STATE  SERVICE

2222/tcp closed EtherNetIP-1

/var/lib/gitea/log is empty

actually, I have just rebooted and now the port is open
PORT STATE SERVICE
2222/tcp open EtherNetIP-1

I can ssh in and get: Permission denied (publickey).

It looks like it’s working I just need to setup the certifcate.