Fix problems with unicode in junit XML output when writing to HTML report

Review Request #4051 — Created July 6, 2016 and submitted

3627, 3628
gmalmquist, molsen, stuhood

When there is unicode in the XML output, using --test-junit-html-report would result in an exception similar to:

File "/Users/zundel/Src/Pants/src/python/pants/backend/jvm/tasks/", line 568, in _execute
  File "/Users/zundel/Src/Pants/src/python/pants/backend/jvm/tasks/", line 562, in _do_report
    html_file_path = JUnitHtmlReport().report(self.workdir, os.path.join(self.workdir, 'reports'))
  File "/Users/zundel/Src/Pants/src/python/pants/backend/jvm/tasks/reports/", line 104, in report

Exception message: 'ascii' codec can't encode character u'\xe5' in position 3508: ordinal not in range(128)

This change converts the encoding to binary before writing it to the file.

Added a unit test that reproduces the problem.

CI green at

  2. This seems like you would want ensure_text? I guess I'm not clear on what ensure_binary is doing here.

    It seems like the html generator should just be encoding the utf-8 characters in the html when it generates the html.

    1. That's what I thought at first too! I'm pretty sure its the write() command that can't handle utf-8 and so this just changes it into a buffer full of bytes.

    2. Oh, ok. So the html is correct? But writing it to the file was failing. Got it.

  1. Ship It!
  1. Hooray for better unicode support!

  2. src/python/pants/util/ (Diff revision 2)

    While you're in here, could you add a test for the error case?

  3. tests/python/pants_test/backend/jvm/tasks/reports/junit_html_report_resources/TEST-org.pantsbuild.UnicodeCharsTest.xml (Diff revision 2)

    Are the properties necessary? I think it'd be clearer without them.

  4. Perhaps rather than asserting the stats, you could assert about the unicode contents? Or do we not handle unicode in source files? Now that would be 悲しい。

Review request changed

Status: Closed (submitted)

Change Summary:

Thanks Chris & Nick. Commit 534eb57.