Better cleaning #8
Conversation
addok_france/utils.py
Outdated
| @@ -50,14 +50,17 @@ | |||
|
|
|||
|
|
|||
| def clean_query(q): | |||
| q = re.sub(r'([\d]{5})', r' \1 ', q, flags=re.IGNORECASE) | |||
| q = re.sub(r'(^| )(boite postale|b\.?p\.?|cs|tsa|cidex) *(n(o|°|) *|)[\d]+ *', r'\1', q, flags=re.IGNORECASE) | |||
There was a problem hiding this comment.
Regex is quite slow, I'm not sure we wanna go that far in cleaning.
Have you had a look on perfs? :)
There was a problem hiding this comment.
regexp are not that slow (I measured 3µs for this one)... and it is called just once to clean the query
addok_france/utils.py
Outdated
| @@ -116,14 +119,17 @@ def flag_housenumber(tokens): | |||
|
|
|||
| def fold_ordinal(s): | |||
| """3bis => 3b.""" | |||
| if s[0].isdigit() and not s.isdigit(): | |||
| if s is not None and s !='' and s[0].isdigit() and not s.isdigit(): | |||
There was a problem hiding this comment.
I feel like this needs to be fixed properly beforehand. I'll have a look.
There was a problem hiding this comment.
No way to reproduce the issue, neither from the shell, the pyshell or the http API.
Can you be a bit more specific on how you get the issue here? A simple way to reproduce from shell or pyshell would help :)
There was a problem hiding this comment.
This needs either a reproducable test case (so we can understand) either removal :)
More variations for BP/CS/CIDEX/TSA like:
B.P 41
TSA N°41
Cleans phone/fax numbers too...