standardize the interface to TransitiveIdeal and friends

[nthiery]

1. Implement a single entry point for recursively enumerated sets:

     RecursivelyEnumeratedSet(seeds, successors, structure=..., enumeration=...)

where structure takes values in the set [None, 'forest', 'graded', 'symmetric'] and enumeration takes values in the set [None, 'depth', 'breadth', 'naive'].

2. Deprecate TransitiveIdeal, TransitiveIdealGraded and SearchForest as entry point.

3. TransitiveIdeal(succ, seeds) keeps the same behavior as before and is now the same as RecursivelyEnumeratedSet(seeds, succ, structure=None, enumeration='naive').

4. TransitiveIdealGraded(succ, seeds, max_depth) keeps the same behavior as before and is now the same as RecursivelyEnumeratedSet(seeds, succ, structure=None, enumeration='breadth', max_depth=max_depth).

Remarks:

A. For now the code of SearchForest is still in sage/combinat/backtrack.py. It should be moved in sage/sets/recursively_enumerated_set.pyx. See #16351.

B. TransitiveIdeal and TransitiveIealGraded are used in the code of sage/combinat, sage/categories and sage/groups at least. These should be updated to use RecursivelyEnumeratedSet for speed improvements and also to avoid issues explained in C below. See #16352.

C. Note that there were some issues with TransitiveIdeal and TransitiveIdealGraded, namely:

• Enumeration of TransitiveIdeal is claimed to be depth first search in the top level file backtrack.py, but in fact, it is neither breadth first neither depth first. It is what I call a naive search.
• Enumeration of TransitiveIdealGraded is indeed breadth first as claimed but it does not make use of the graded hypothesis at all because it remembers every generated elements.

See my status report at SageDays57 for more info.

Depends on #14052

CC: @sagetrac-sage-combinat @hivert

Component: combinatorics

Keywords: backtrack, enumerated set, transitive closure, days57

Author: Sébastien Labbé

Branch: 3191690

Reviewer: Travis Scrimshaw

Issue created by migration from https://trac.sagemath.org/ticket/6637

[seblabbe]

comment:1

Here are some discussions related to this ticket:

• TransitiveIdeal vs TransitiveIdealGraded on sage-combinat, Feb 2013.

• Comment of Nicolas Thiéry at #14052, Feb 2013

• SearchForest and post_process..., on sage-devel, Oct. 2012.

• Sage modules and forking on sage-devel, a comment by Florent Hivert, Oct. 2012.

• Parallel map reduce on SearchForest #13580, patch available on sage-combinat : trac_13580-map_reduce-fh.patch

[seblabbe]

Description changed:

--- 
+++ 
@@ -4,6 +4,6 @@
      TransitiveClosure(roots, operators = ..., children = , acyclic = True, algo = "DFS", "BFS", internal_nodes = False)
 ```
 
-for all the functions in sage.combinat.backtrack.py SearchForest, TransitiveIdeal, TransitiveIdealGraded
+for all the functions in sage.combinat.backtrack.py `SearchForest`, `TransitiveIdeal`, `TransitiveIdealGraded`
 
 TODO: discuss the names above

[seblabbe]

comment:4

In a tentative to find good choices for name and so on, here is how I see this.

• Let X be a set.
• Let R be a binary relation on X, that is R is a subset of the cartesian product of X times X.
• Denote by xRy if x is R-related to y.

Now

• Let seeds be a subset of X.
• Let succ be a callable python object X -> 2^X such that xRy if and only if y is in succ(x).

We are interested in the subset S of X that can be generated from the seeds using the succ recursively. More precisely in the set

S = {y : x = x1 R x2 R x3 R ... R xn = y and x in seeds}.

Moreover, we are interested in the enumeration of S itself and we consider depth-first and breadth-first as different and both usefull.

Such a relation G = (X,R) can be seen as a directed graph. I think this remark is useful as it may provide some vocabulary. Indeed the set S is the connected components of the generators in the digraph G.

I see some different cases :

1. We do not know anything more about the relation. We need to save in memory all the known objects to avoid duplicates. This is what is currently done in TransitiveIdeal (depth-first search) and (curiously) in TransitiveIdealGraded (breadth-first search) also.

2. The directed graph S is a forest with given seeds. Equivalently, one may say that S do not contain cycle (oriented or not). This is what is currently done in SearchForest.

3. The relation is graded. By graded here I mean what I thought TransitiveIdealGraded was doing until I look more carefully at the doc and the code. More seriously, by graded I mean the following : for all (x1 in seeds) and (y1 in seeds),

if (x1 R x2 R ... R xn) and (y1 R y2 R ... R ym) and (xn=ym), then (n=m).

The relation is graded if all path from the origin to an element have the same length. In this case, we only need to save in memory the current level.

4. The relation is symmetric. If the relation is symmetric, we only need to keep in memory the last two level of depth. This is what I needed and coded this week. And this is why I started to look more carefully at the code in sage/combinat/backtrack.py...

That is it for now!

[nthiery]

comment:5

I totally agree with the analysis!

I don't know yet what would be the best name for the argument provided by the user to describe the relation. Behind the scene we are definitely modelling relation. But what the user provide is not the relation but a function that computes the (out) neighbors for this relation. If at the end of the day we choose "TransitiveClosure" as name for the main entry point, then "neighbors" would be consistent. If we go for "RecursiveSet" (or RecursiveEnumeratedSet or variant thereof) then "operators" would be consistent.

Cheers,
Nicolas

[seblabbe]

comment:6

I think relation would not be too bad for the name of the successor keyword and should be considered as well. In some sense, a binary relation is a function f: X -> 2^X and a function is a relation such that |f(x)|=1 for all x.

Possibilities for successor keyword :

• succ : suitable for non symmetric relation, currently used in TransitiveIdeal and TransitiveIdealGraded
• successors
• operators
• neighbors : suitable vocabulary for symmetric relation
• children : suitable vocabulary for non symmetric relation, currently used in SearchForest.
• relatives : suitable vocabulary for symmetric relation

Possibilities for generators keyword :

• generators
• gens
• roots
• seeds

[seblabbe]

comment:7

If we go for RecursiveSet (or RecursiveEnumeratedSet or variant thereof)

I like RecursiveSet. Maybe RecursiveEnumeratedSet is more related to what we do but is also longer.

Some links:

• http://en.wikipedia.org/wiki/Recursive_set
• http://en.wikipedia.org/wiki/Recursively_enumerable_set

[seblabbe]

comment:8

I don't know yet what would be the best name for the argument provided by the user to describe the relation.

For the above 4 cases, I would suggest arguments like the following :

RecursiveSet(seeds, succ)
RecursiveSet(seeds, succ, structure="forest")
RecursiveSet(seeds, succ, structure="graded")
RecursiveSet(seeds, succ, structure="symmetric")

[seblabbe]

Attachment: trac_6637_recursive_set-sl.patch.gz

[seblabbe]

comment:9

I just added a patch. It implements the RecursiveSet_symmetric class and creates factory called RecursiveSet. For now, RecursiveSet returns either an instance of TransitiveIdeal, SearchForest or RecursiveSet_symmetric. I started an empty class RecursiveSet_graded. See examples inside the docstring of the class RecursiveSet.

It is not ready for review, but comments are welcome to help me continue this work.

Actually, my questions are :

• How should I merge RecursiveSet with TransitiveIdeal and SearchForest?
• Do we like this interface?