avoid pathological regex performance when linkifying large ivy output

Review Request #3603 - Created March 24, 2016 and submitted

Information
Matt Landis
pants
3085
Reviewers
pants-reviews
kwlzn, stuhood, zundel

Our build contains an 800 MB jar. Ivy outputs 110k dots while downloading this. Pants then tries to linkify this and re.findall takes 10+ minutes.

I tried instead changing all the groups to non-capturing and all the ? and +s to non-greedy, but they didn't solve the problem (they did help a bit, but not enough to suffice).

The negative lookahead solution in this commit feels hacky. I'm open to other suggestions (it might be worth trying the regex or re2 libraries for non-backtracking regexes, if it's worth adding those deps to pants - also not sure if those will actually help, since we'll still be quadratic).

https://travis-ci.org/pantsbuild/pants/builds/118159590

also turned linkify.py into a standalone script to repro the issue on our ivy output and verified execution time went from awful -> good

Kris Wilson
Stu Hood
Benjy Weinberger
Kris Wilson
Timur Abishev
Matt Landis
Review request changed

Status: Closed (submitted)

Loading...