无法转换 org.apache.hadoop.fs.FileSystem$Cache。 org.apache.hadoop.fs.FileSystem$Cache$Key 类被冻结

Question

我试图在我的 scala 测试中模拟 hadoop 文件系统。任何想法如何解决这个问题请：

import java.net.URI
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.junit.Test
import org.junit.runner.RunWith
import org.mockito.ArgumentMatchers.any
import org.mockito.Mockito._
import org.powermock.api.mockito.PowerMockito
import org.powermock.core.classloader.annotations.PrepareForTest
import org.powermock.modules.junit4.PowerMockRunner

@RunWith(classOf[PowerMockRunner])
@PrepareForTest(Array(classOf[FileSystem]))  
class SimpleFileSystemTest {

  @Test
  def testMockStaticFileSystem(): Unit = {
    // Mock static method FileSystem.get()
    PowerMockito.mockStatic(classOf[FileSystem])
    val mockFileSystem = mock(classOf[FileSystem])

    // Define behavior of FileSystem.get()
    when(FileSystem.get(any(classOf[URI]), any(classOf[Configuration]))).thenReturn(mockFileSystem)

    // Add simple assertions to verify the behavior
    val uri = new URI("s3a://bucket")
    val conf = new Configuration()

    // Verify that the mocked FileSystem is returned
    val fileSystem = FileSystem.get(uri, conf)
    assert(fileSystem eq mockFileSystem)

    // Verify that the static method was called
    PowerMockito.verifyStatic(classOf[FileSystem], times(1))
    FileSystem.get(uri, conf)
  }
}

但是我得到了这个例外：

org.mockito.exceptions.base.MockitoException: 
Mockito cannot mock this class: class org.apache.hadoop.fs.FileSystem.

Mockito can only mock non-private & non-final classes.
If you're not sure why you're getting this error, please report to the mailing list.


Java               : 1.8
JVM vendor name    : Amazon.com Inc.
JVM vendor version : 25.422-b05
JVM name           : OpenJDK 64-Bit Server VM
JVM version        : 1.8.0_422-b05
JVM info           : mixed mode
OS name            : Mac OS X
OS version         : 14.7


Underlying exception : java.lang.IllegalStateException: Failed to transform class with name org.apache.hadoop.fs.FileSystem$Cache. Reason: org.apache.hadoop.fs.FileSystem$Cache$Key class is frozen

    at SimpleFileSystemTest.testMockStaticFileSystem(SimpleFileSystemTest.scala:19)
    at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:68)
    at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:89)
    at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:97)
    at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:87)
    at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:50)
    at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
    at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
    at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.IllegalStateException: Failed to transform class with name org.apache.hadoop.fs.FileSystem$Cache. Reason: org.apache.hadoop.fs.FileSystem$Cache$Key class is frozen
    at org.powermock.core.classloader.javassist.JavassistMockClassLoader.defineAndTransformClass(JavassistMockClassLoader.java:119)
    at org.powermock.core.classloader.MockClassLoader.loadMockClass(MockClassLoader.java:174)
    at org.powermock.core.classloader.MockClassLoader.loadClassByThisClassLoader(MockClassLoader.java:102)
    at org.powermock.core.classloader.DeferSupportingClassLoader.loadClass1(DeferSupportingClassLoader.java:147)
    at org.powermock.core.classloader.DeferSupportingClassLoader.loadClass(DeferSupportingClassLoader.java:98)

这些是我的依赖项：

libraryDependencies ++= Seq(
      "org.slf4j" % "slf4j-api" % "2.0.7" % "provided",
      "com.google.guava" % "guava" % "32.1.2-jre" % "test",
      "org.scalatest" %% "scalatest" % "3.2.16" % "test",
      "org.scalatestplus" %% "mockito-3-4" % "3.2.10.0" % "test",
      "org.powermock" % "powermock-module-junit4" % "2.0.9" % "test",
      "org.powermock" % "powermock-api-mockito2" % "2.0.9" % "test" exclude("org.mockito", "mockito-core"),
      "org.mockito" % "mockito-core" % "3.12.4" % "test",

Answer 1

1. 您正在尝试模拟文件系统，它是抽象的。您需要模拟它的子类，例如

   RawLocalFileSystem

   。考虑一下那个班级，稳定。

2. 避免嘲笑 S3AFS，因为它很复杂，并且以甚至破坏其自己的模拟测试的方式移动其内部结构。考虑该类的内部方法不稳定。

3. 有一个包私有静态方法

   FileSystem.addFileSystemForTesting()

   ，它可以让你将任何 FS 子类推入缓存，并在 get() 调用时返回它。您需要在包中添加一些类才能获得它； hadoop-common 测试 JAR 有一个类

   FileSystemTestHelper

   ，它提供从包外部对其的公共访问。考虑这个稳定。

4. 如果您真的非常想要/需要子类化 S3A FS，hadoop-aws 测试 JAR 有一个类

   org.apache.hadoop.fs.s3a.MockS3AFileSystem

   ，它实际上是 S3AFS 的子类；它将调用中继到（通常是mockito）模拟的 fs，您可以在其中调整公共方法，而不必担心 s3a 代码更改的大部分复杂性。但这不供公众使用，并且没有稳定性保证。如果更新破坏了您的代码：您需要做的作业来修复该问题。