Notes on add password to existing ssh-rsa key, can not connect to EC2 using private key, using supervisord on Amazon EC2 etc.

https://stackoverflow.com/questions/3818886/how-do-i-add-a-password-to-an-openssh-private-key-that-was-generated-without-a-p

Supervisord
https://serverfault.com/questions/672891/supervisor-setup-on-aws-ami-linux
https://security.stackexchange.com/questions/59136/can-i-add-a-password-to-an-existing-private-key

This one work for me, so I do not try another method
openssl rsa -aes256 -in your.key -out your.encrypted.key
mv your.encrypted.key your.key
chmod 600 your.key

Damn my bone head
I know that Laravel queue require beanstalkd and or supervisord or similar to automatically start queue. But when investigate old server for upgrade, I only focus on beanstalkd. So now I have to migrate supervisord configurations after sever roll out to production.

Config here
http://supervisord.org/configuration.html

https://stackoverflow.com/questions/1968773/view-stored-procedure-function-definition-in-mysql

https://stackoverflow.com/questions/51908004/install-phpredis-mac-osx

supervisorctl reread
supervisorctl update

at the end
sudo /etc/init.d/supervisord start did the trick

not sure what happened but may be this command not stop so I ctrl-C

Yeah, one more time my bone head keep forgot where log to digging. This time it is /var/log/message for supervisor:
$tail -f /var/log/message
supervisord: Starting supervisord: Error: The directory named as part of the path /var/run/supervisord/supervisor.pid does not exist

I have run $find /var/log -mmin -3 (find all log recently updated) but ignore file message.

systemd: PID file /var/run/supervisord.pid not readable (yet?) after start.
 my config is different with this default or /var/run/supervisor.pid, so may be it caused the problem?
 ;pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
pidfile=/var/run/supervisord/supervisor.pid ;

 [keepadmin@ip-10-0-1-218 ~]$ sudo service supervisord restart
Restarting supervisord (via systemctl):  Job for supervisord.service failed because a timeout was exceeded. See "systemctl status supervisord.service" and "journalctl -xe" for details.
                                                           [FAILED]

Even with supervisord failed to restart, queue still work, may be it kick started and keep running, but it may be not work on restart server. This may explain why my supervisord and queue not autostart on reboot.

Try update config to 
/var/run/supervisord.pid

 systemd: PID file /var/run/supervisord.pid not readable (yet?) after start.

https://www.howtogeek.com/168156/fixing-authentication-refused-bad-ownership-or-modes-for-directory/

Authentication refused: bad ownership or modes for directory /home/fuckinguser

tail -f /var/log/secure (CentOS)
sshd[5527]: Authentication refused: bad ownership or modes for directory /home/user2/.ssh
sshd[5527]: Connection closed by 118.70.133.151 port 54221 [preauth]
sshd[5545]: Authentication refused: bad ownership or modes for directory /home/user2/.ssh
sshd[5545]: Connection closed by 118.70.133.151 port 54236 [preauth]

And don't forget to double check EC2 Security Group Inbound rules (allow IP)

SSH debug
ssh -vvv -i ~/your_key.pem user2@SERVER_IP

=> my error log from client:
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.

=> After long time debugging, I realize that I should see log on Server side (Luckily I have another account for login SSH).