Skip to content
This repository was archived by the owner on Mar 9, 2023. It is now read-only.

Commit

Permalink
Fix connection cost lookup, fix #129 (#131)
Browse files Browse the repository at this point in the history
* Fix connection cost lookup, fix #129

The connection costs lookup was backwards.

There was a comment in the pre-cython code that the call to the cost
lookup function looked backwards, but was actually correct. It was
calling with (l_node.right_id, r_node.left_id). I kept this order when I
replaced the function call with a memoryview access, but that was wrong;
I hadn't noticed that the access function was actually reversing the
order of its arguments.

This fix was verified by checking the tokenization of a short document
vs v0.4.5 and making sure there were no changes.

* Remove old comment
  • Loading branch information
polm authored Jun 18, 2020
1 parent 1c4ba98 commit 6276fa9
Showing 1 changed file with 1 addition and 2 deletions.
3 changes: 1 addition & 2 deletions sudachipy/lattice.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,7 @@ cdef class Lattice:
for l_node in self.end_lists[begin]:
if not l_node.is_connected_to_bos:
continue
# right_id and left_id look reversed, but it works ...
connect_cost = self.connect_costs[l_node.right_id, r_node.left_id]
connect_cost = self.connect_costs[r_node.left_id, l_node.right_id]

# 0x7fff == Grammar.INHIBITED_CONNECTION:
if connect_cost == 0x7fff:
Expand Down

0 comments on commit 6276fa9

Please sign in to comment.