当前位置:   article > 正文

spark 自定义partitioner_spark-shell 找不到partitioner

spark-shell 找不到partitioner
package inpv

import org.apache.spark.Partitioner

class PvPartitioner(numParts: Int) extends Partitioner {
    override def numPartitions: Int = numParts

    override def getPartition(key: Any): Int = {
        val code = (key.toString().hashCode % numPartitions)
        if (code < 0) {
            code + numPartitions
        } else {
            code
        }
    }

    override def equals(other: Any): Boolean = other match {
        case p: PvPartitioner =>
            p.numPartitions == numPartitions
        case _ =>
            false
    }

    override def hashCode: Int = numPartitions
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/2023面试高手/article/detail/727490
推荐阅读
相关标签
  

闽ICP备14008679号