This article explains an issue with CSO 4.1.1 where the getlogs script is not working and the workaround that can be used.
getlogs
Output of getlogs script:
# ./getlogs.sh Log collection for csp-central-msvm2 - X.X.X.X [+ ] 0% csp.admin-portal-ui-4262910192-0qs6lError from server (NotFound): the server could not find the requested resource ( pods/log csp.admin-portal-ui-4262910192-0qs6l) [++ ] 0% csp.admin-portal-ui-4262910192-7p2lcError from server (NotFound): the server could not find the requested resource ( pods/log csp.admin-portal-ui-4262910192-7p2lc) [+++ ] 1% csp.admin-portal-ui-4262910192-cb1d6Error from server (NotFound): the server could not find the requested resource ( pods/log csp.admin-portal-ui-4262910192-cb1d6) [++++ ] 1% csp.csp-ams-2930958688-lnj23Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-ams-2930958688-lnj23) [+++++ ] 2% csp.csp-ams-2930958688-rggcsError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-ams-2930958688-rggcs) [++++++ ] 2% csp.csp-ams-2930958688-sm9mcError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-ams-2930958688-sm9mc) [+++++++ ] 3% csp.csp-cslm-signature-ims-central-3520248340-7qlvmError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-cent ral-3520248340-7qlvm) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-7qlvm) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-7qlvm) [++++++++ ] 3% csp.csp-cslm-signature-ims-central-3520248340-h0x8nError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-cent ral-3520248340-h0x8n) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-h0x8n) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-h0x8n) [+++++++++ ] 4% csp.csp-cslm-signature-ims-central-3520248340-rkspzError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-cent ral-3520248340-rkspz) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-rkspz) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-rkspz) [++++++++++ ] 4% csp.csp-cslm-signature-ims-central-core-1107922945-8gg43Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims -central-core-1107922945-8gg43) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-8gg43) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-8gg43) [+++++++++++ ] 5% csp.csp-cslm-signature-ims-central-core-1107922945-jr899Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims -central-core-1107922945-jr899) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-jr899) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-jr899) [++++++++++++ ] 5% csp.csp-cslm-signature-ims-central-core-1107922945-tkppwError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims -central-core-1107922945-tkppw) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-tkppw) Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-tkppw)
This issue is due to kubectl logs being blocked in CSO 4.1.1 (refer to https://contrail-jws.atlassian.net/browse/CXU-36343 for complete details).
Commit was done with the following changes to control the kubelet log size:
The verbosity level for the log was reduced from 6 to 2. This prevents kubelet from aggressively writing logs, and the log file size will then be in control. To see this change in the customer setup, locate the file /etc/default/kubelet in MSVM. Then change the following lines:
/etc/default/kubelet
Before: ---enable-debugging-handlers=true \ Change To: ---enable-debugging-handlers=false \ Before: ---v=6 \ Change To: ---v=2 \
Before: ---enable-debugging-handlers=true \
---enable-debugging-handlers=true \
Change To: ---enable-debugging-handlers=false \
---enable-debugging-handlers=false \
Before: ---v=6 \
---v=6 \
Change To: ---v=2 \
---v=2 \
Restart kubelet to have these changes take effect: service kubelet restart
service kubelet restart
However, after this change, kubectl logs <pod> will not be present. But docker logs and Kibana will continue to work.
kubectl logs <pod>
There are two workarounds, and any one of them can be used:
This workaround can be used to collect logs and should be reverted in order to use disk space properly.
Note: Be aware that the kubectl logs that are consuming most of the disk space are reported in https://contrail-jws.atlassian.net/browse/CXU-36343 after enabling kubectl logs.
vi /etc/default/kubelet file in all msvms, 4.1.1 FRS: ---enable-debugging-handlers=false \ Change To: ---enable-debugging-handlers=true \ 4.1.1 FRS: ---v=2 \ Change To: ---v=6 \ service kubelet restart
Download the archive that contains getdockerlogs.sh and getlogs.sh scripts from here . Then extract the zip files into your workstation. It contains two scripts: getdockerlogs.sh and getlogs.sh .
getdockerlogs.sh
getlogs.sh
Below are the instructions:
sed -i -e 's/\r$//' getlogs.sh
/home/Contrail_Service_Orchestration_4.1.1/deployments/central/file_root/kubeminion/files
${DATE.EN_US}_ms_logs.tgz
Snippet:
[email protected]:/home/ubuntu/Contrail_Service_Orchestration_4.1.1# bash getlogs.sh Running docker logs in kubeminion VMs... Pushing docker logs to installer VM... Compressing the Logs... ***** All the logs are collected under 09-20-19-10-38-50_ms_logs.tgz****** root@ip-100-74-0-52:/home/ubuntu/Contrail_Service_Orchestration_4.1.1# ls -la total 204808 drwxr-xr-x 33 root root 4096 Sep 20 10:39 . drwxr-xr-x 6 ubuntu ubuntu 4096 Aug 15 13:44 .. -rw-r--r-- 1 root root 209290644 Sep 20 10:39 09-20-19-10-38-50_ms_logs.tgz <- LOG FILE drwxr-xr-x 3 root root 4096 Aug 14 11:31 apache_deb_packages
Note: It is recommended to delete the log files after usage to avoid getting the disk full.
2019-10-14: Removed internal and added docker workaround
2019-11-12: Added comment about DOS format bash precaution in Step 2 > Workaround 2 in Solution