class AggregateNeighbors extends Serializable with Logging with WithIntermediateStorageLevel with WithCheckpointInterval with WithLocalCheckpoints

A class for performing multi-hop neighbor aggregation on a graph.

AggregateNeighbors allows you to explore the graph up to a specified number of hops, accumulating values along paths using customizable accumulator expressions. It supports both stopping conditions (when to stop exploring) and target conditions (when to collect a result). If no target condition is provided, collect all traversals that reached stopping condition. The algorithm processes the graph in a breadth‑first manner.

Use the builder pattern to configure parameters, then call run() to execute.

Linear Supertypes
WithLocalCheckpoints, WithCheckpointInterval, WithIntermediateStorageLevel, Logging, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AggregateNeighbors
  2. WithLocalCheckpoints
  3. WithCheckpointInterval
  4. WithIntermediateStorageLevel
  5. Logging
  6. Serializable
  7. AnyRef
  8. Any
Implicitly
  1. by any2stringadd
  2. by StringFormat
  3. by Ensuring
  4. by ArrowAssoc
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. def +(other: String): String
    Implicit
    This member is added by an implicit conversion from AggregateNeighbors toany2stringadd[AggregateNeighbors] performed by method any2stringadd in scala.Predef.
    Definition Classes
    any2stringadd
  4. def ->[B](y: B): (AggregateNeighbors, B)
    Implicit
    This member is added by an implicit conversion from AggregateNeighbors toArrowAssoc[AggregateNeighbors] performed by method ArrowAssoc in scala.Predef.
    Definition Classes
    ArrowAssoc
    Annotations
    @inline()
  5. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  6. def addAccumulator(name: String, init: Column, update: Column): AggregateNeighbors.this.type

    Adds a single accumulator to those already configured.

    Adds a single accumulator to those already configured.

    This method can be called multiple times to add several accumulators.

    name

    accumulator column name

    init

    Column expression for the accumulator's initial value

    update

    Column expression that updates the accumulator when traversing an edge

    returns

    this AggregateNeighbors instance for method chaining

  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. val checkpointInterval: Int
    Attributes
    protected
    Definition Classes
    WithCheckpointInterval
  9. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  10. def ensuring(cond: (AggregateNeighbors) => Boolean, msg: => Any): AggregateNeighbors
    Implicit
    This member is added by an implicit conversion from AggregateNeighbors toEnsuring[AggregateNeighbors] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  11. def ensuring(cond: (AggregateNeighbors) => Boolean): AggregateNeighbors
    Implicit
    This member is added by an implicit conversion from AggregateNeighbors toEnsuring[AggregateNeighbors] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  12. def ensuring(cond: Boolean, msg: => Any): AggregateNeighbors
    Implicit
    This member is added by an implicit conversion from AggregateNeighbors toEnsuring[AggregateNeighbors] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  13. def ensuring(cond: Boolean): AggregateNeighbors
    Implicit
    This member is added by an implicit conversion from AggregateNeighbors toEnsuring[AggregateNeighbors] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  14. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  16. def getCheckpointInterval: Int

    Gets checkpoint interval.

    Gets checkpoint interval.

    Definition Classes
    WithCheckpointInterval
  17. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  18. def getIntermediateStorageLevel: StorageLevel

    Gets storage level for intermediate datasets that require multiple passes.

    Gets storage level for intermediate datasets that require multiple passes.

    Definition Classes
    WithIntermediateStorageLevel
  19. def getUseLocalCheckpoints: Boolean

    Gets whether local checkpoints are being used instead of regular checkpoints.

    Gets whether local checkpoints are being used instead of regular checkpoints.

    returns

    true if local checkpoints are enabled, false otherwise

    Definition Classes
    WithLocalCheckpoints
  20. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  21. val intermediateStorageLevel: StorageLevel
    Attributes
    protected
    Definition Classes
    WithIntermediateStorageLevel
  22. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  23. def logDebug(s: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logInfo(s: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logTrace(s: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logWarn(s: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  28. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  29. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  30. def resultIsPersistent(): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def run(): DataFrame

    Executes the configured neighbor aggregation and returns the result DataFrame.

    Executes the configured neighbor aggregation and returns the result DataFrame.

    The result contains one row per (starting vertex, target vertex) pair that satisfied either the stopping condition or the target condition. Columns are:

    • id: the vertex ID of the target (or stopping) vertex
    • hop: the number of hops taken to reach it
    • one column for each accumulator, holding its final value
    returns

    DataFrame with aggregation results

  32. def setAccumulators(names: Seq[String], inits: Seq[Column], updates: Seq[Column]): AggregateNeighbors.this.type

    Defines multiple accumulators used to compute values along each path.

    Defines multiple accumulators used to compute values along each path.

    Each accumulator has a name, an initial value expression, and an update expression. The update expression is evaluated at each hop and can refer to:

    • the accumulator's previous value (using the accumulator's column name)
    • source vertex attributes via srcAttr("attrName")
    • destination vertex attributes via dstAttr("attrName")
    • edge attributes via edgeAttr("attrName")
    names

    sequence of accumulator column names

    inits

    sequence of Column expressions that give the initial value for each accumulator (evaluated on starting vertices)

    updates

    sequence of Column expressions that define how to update the accumulator when traversing an edge

    returns

    this AggregateNeighbors instance for method chaining

  33. def setCheckpointInterval(value: Int): AggregateNeighbors.this.type

    Sets checkpoint interval in terms of number of iterations (default: 2).

    Sets checkpoint interval in terms of number of iterations (default: 2). Checkpointing regularly helps recover from failures, clean shuffle files, shorten the lineage of the computation graph, and reduce the complexity of plan optimization. As of Spark 2.0, the complexity of plan optimization would grow exponentially without checkpointing. Hence, disabling or setting longer-than-default checkpoint intervals are not recommended. Checkpoint data is saved under org.apache.spark.SparkContext.getCheckpointDir with prefix of the algorithm name. If the checkpoint directory is not set, this throws a java.io.IOException. Set a nonpositive value to disable checkpointing. This parameter is only used when the algorithm is set to "graphframes". Its default value might change in the future.

    Definition Classes
    WithCheckpointInterval
    See also

    org.apache.spark.SparkContext.setCheckpointDir in Spark API doc

  34. def setEdgeFilter(value: Column): AggregateNeighbors.this.type

    Filters which edges can be traversed during the aggregation.

    Filters which edges can be traversed during the aggregation.

    Only edges satisfying the given column expression are considered. The default is true (all edges are traversable).

    value

    a Boolean Column expression that filters edges

    returns

    this AggregateNeighbors instance for method chaining

  35. def setIntermediateStorageLevel(value: StorageLevel): AggregateNeighbors.this.type

    Sets storage level for intermediate datasets that require multiple passes (default: MEMORY_AND_DISK).

    Sets storage level for intermediate datasets that require multiple passes (default: MEMORY_AND_DISK).

    Definition Classes
    WithIntermediateStorageLevel
  36. def setMaxHops(value: Int): AggregateNeighbors.this.type

    Sets the maximum number of hops to explore from the starting vertices.

    Sets the maximum number of hops to explore from the starting vertices.

    The algorithm will stop after this many iterations even if the stopping condition hasn't been reached for all paths. The default is 3. Be aware that high value of max hops on a dense graph will lead to a huge memory load and potential OOM errors.

    value

    positive integer for maximum hop count

    returns

    this AggregateNeighbors instance for method chaining

  37. def setRemoveLoops(value: Boolean): AggregateNeighbors.this.type

    Controls whether self‑loops (edges where src == dst) are excluded.

    Controls whether self‑loops (edges where src == dst) are excluded.

    value

    if true, self‑loop edges are filtered out

    returns

    this AggregateNeighbors instance for method chaining

  38. def setRequiredEdgeAttributes(values: Seq[String]): AggregateNeighbors.this.type

    Specifies which edge attributes should be carried through the traversal.

    Specifies which edge attributes should be carried through the traversal.

    By default, all edge columns are carried. Specifying a subset can improve performance by reducing the amount of data shuffled.

    values

    sequence of edge column names to keep

    returns

    this AggregateNeighbors instance for method chaining

  39. def setRequiredVertexAttributes(values: Seq[String]): AggregateNeighbors.this.type

    Specifies which vertex attributes should be carried through the traversal.

    Specifies which vertex attributes should be carried through the traversal.

    By default, all vertex columns are carried. Specifying a subset can improve performance by reducing the amount of data shuffled.

    values

    sequence of vertex column names to keep

    returns

    this AggregateNeighbors instance for method chaining

  40. def setStartingVertices(value: Column): AggregateNeighbors.this.type

    Specifies which vertices to start the aggregation from.

    Specifies which vertices to start the aggregation from.

    Only vertices satisfying the given column expression will be used as seeds for the traversal. The default is true (all vertices are seeds).

    value

    a Boolean Column expression that selects seed vertices

    returns

    this AggregateNeighbors instance for method chaining

  41. def setStoppingCondition(value: Column): AggregateNeighbors.this.type

    Sets a condition that, when true for a vertex, stops further exploration along that path.

    Sets a condition that, when true for a vertex, stops further exploration along that path.

    The column expression can refer to:

    • accumulator columns
    • source vertex attributes via srcAttr("attrName")
    • destination vertex attributes via dstAttr("attrName")
    • edge attributes via edgeAttr("attrName")

    Either setStoppingCondition or setTargetCondition must be called. When both are provided only accumulators that reach targetCondition are saved to results.

    value

    a Boolean Column expression that triggers stopping

    returns

    this AggregateNeighbors instance for method chaining

  42. def setTargetCondition(value: Column): AggregateNeighbors.this.type

    Sets a condition that, when true for a vertex, marks it as a target and saves the accumulator values.

    Sets a condition that, when true for a vertex, marks it as a target and saves the accumulator values.

    Either setStoppingCondition or setTargetCondition must be called. When both are provided only accumulators that reach targetCondition are saved to results.

    value

    a Boolean Column expression that defines a target vertex

    returns

    this AggregateNeighbors instance for method chaining

  43. def setUseLocalCheckpoints(value: Boolean): AggregateNeighbors.this.type

    Sets whether to use local checkpoints instead of regular checkpoints (default: false).

    Sets whether to use local checkpoints instead of regular checkpoints (default: false). Local checkpoints are faster but less reliable as they don't survive node failures.

    value

    true to use local checkpoints, false for regular checkpoints

    returns

    this instance

    Definition Classes
    WithLocalCheckpoints
  44. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  45. def toString(): String
    Definition Classes
    AnyRef → Any
  46. val useLocalCheckpoints: Boolean
    Attributes
    protected
    Definition Classes
    WithLocalCheckpoints
  47. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  48. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  49. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

  2. def formatted(fmtstr: String): String
    Implicit
    This member is added by an implicit conversion from AggregateNeighbors toStringFormat[AggregateNeighbors] performed by method StringFormat in scala.Predef.
    Definition Classes
    StringFormat
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.12.16) Use formatString.format(value) instead of value.formatted(formatString), or use the f"" string interpolator. In Java 15 and later, formatted resolves to the new method in String which has reversed parameters.

  3. def [B](y: B): (AggregateNeighbors, B)
    Implicit
    This member is added by an implicit conversion from AggregateNeighbors toArrowAssoc[AggregateNeighbors] performed by method ArrowAssoc in scala.Predef.
    Definition Classes
    ArrowAssoc
    Annotations
    @deprecated
    Deprecated

    (Since version 2.13.0) Use -> instead. If you still wish to display it as one character, consider using a font with programming ligatures such as Fira Code.

Inherited from WithLocalCheckpoints

Inherited from WithCheckpointInterval

Inherited from WithIntermediateStorageLevel

Inherited from Logging

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Inherited by implicit conversion any2stringadd fromAggregateNeighbors to any2stringadd[AggregateNeighbors]

Inherited by implicit conversion StringFormat fromAggregateNeighbors to StringFormat[AggregateNeighbors]

Inherited by implicit conversion Ensuring fromAggregateNeighbors to Ensuring[AggregateNeighbors]

Inherited by implicit conversion ArrowAssoc fromAggregateNeighbors to ArrowAssoc[AggregateNeighbors]

Ungrouped