Fix: Exception when parsing log #20966 (#23301)
authorJakub Novák <kubus.novak@gmail.com>
Tue, 10 May 2022 20:43:25 +0000 (22:43 +0200)
committerGitHub <noreply@github.com>
Tue, 10 May 2022 20:43:25 +0000 (22:43 +0200)
commit863b2576423e1a7933750b297a9b4518ae598db9
treec15fdda7c2ae72e4598e7831d6a68bc5b0801ba9
parentc5b72bf30c8b80b6c022055834fc7272a1a44526
Fix: Exception when parsing log #20966 (#23301)

* UnicodeDecodeError: 'utf-8' codec can't decode byte 0xXX in position X: invalid start byte

  File "/opt/work/python395/lib/python3.9/site-packages/airflow/hooks/subprocess.py", line 89, in run_command
    line = raw_line.decode(output_encoding).rstrip()            # raw_line ==  b'\x00\x00\x00\x11\xa9\x01\n'
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 4: invalid start byte

* Update subprocess.py

* Update subprocess.py

* Fix:  Exception when parsing log #20966

* Fix:  Exception when parsing log #20966

 Another alternative is: try-catch it.

e.g.

```
            line = ''
            for raw_line in iter(self.sub_process.stdout.readline, b''):
                try:
                    line = raw_line.decode(output_encoding).rstrip()
                except UnicodeDecodeError as err:
                    print(err, output_encoding, raw_line)
                self.log.info("%s", line)
```

* Create test_subprocess.sh

* Update test_subprocess.py

* Added shell directive and license to test_subprocess.sh

* Distinguish between raw and decoded lines as suggested by @uranusjr

* simplify test

Co-authored-by: muhua <microhuang@live.com>
airflow/hooks/subprocess.py
airflow/providers/cncf/kubernetes/utils/pod_manager.py
tests/hooks/test_subprocess.py