如何在二进制序列化数据中向avro4s添加演化

Question

我有一个这样的a类：

case class SurveyTemplate(id:UUID,
                          name: String,
                          version: Int,
                          preDefinedChapter: Seq[Chapter]) {
}

我用这个把它序列化为DAO中的二进制代码。

def save(input: SurveyTemplate) = Future{
    val target = toFile(input)
    val output = AvroOutputStream.binary[SurveyTemplate](target)
    output.write(input)
    output.flush()
    output.close()
    input
  }

现在我将二进制文件保存到硬盘上。一切都好。现在我想用"dummy: SurveyTemplate“这样的新字段更改字符串。

如果我现在加载文件并尝试反序列化，我会得到一个java.util.NoSuchElementException，这意味着它们不是可以反序列化的文件。如果我删除额外的字段，一切都会正常工作。

下面是list方法：

def list(name: Option[String] = None,
                    version: Option[Int] = None,
                    authorName: Option[String] = None,
                    drop: Int = 0,
                    take: Int = 100) = Future{
    storageDirectory.listFiles(new FilenameFilter {
      override def accept(dir: File, name: String): Boolean = name.endsWith(ending)
    }).map{e =>
      println("FOUND " + e.getName)    
      val i = AvroInputStream.binary[SurveyTemplate](e)
      println("BINARY " + i)
      println(i.records.size())
      println( Try(i.iterator().toSeq.head) )
      val p = i.iterator().toSeq.head
      i.close()
      println(p)
      println("------------------")
      Some(p)
    }.filter(_.isDefined).slice(drop,take).map(_.get)
  }

我如何才能增加进化的可能性？

谢谢。

添加:打印结果

BINARY com.sksamuel.avro4s.AvroBinaryInputStream@58d20d47
0
Failure(java.util.NoSuchElementException: head of empty stream)

我的新SurveyTemplate：

case class SurveyTemplate(id:UUID,
                          name: String,
                          version: Int,
                          preDefinedChapter: Seq[Chapter], 
                          dummy: Option[String] = None) {
}

下面是最小的

import java.io.{File, FilenameFilter}
import java.util.UUID

import com.sksamuel.avro4s.{AvroInputStream, AvroOutputStream, AvroSchema, SchemaFor}

import scala.util.Try

object Main {

  def main(args: Array[String]): Unit ={
    val dao = new SurveyTemplateDAO
    //RUN THIS FIRST!
    val first = SurveyTemplate(UUID.randomUUID(),"TEST",0,"ICH",Seq(Chapter(1,1,"HELLO",Nil)))
    dao.save(first)
    //COMMENT OUT! Then uncomment the dummy in SurveyTemplate

    //RUN THIS AFTER FILE SAVED
    dao.list()

  }

}


case class Chapter(counter: Int,
                   level: Int,
                   title: String,
                   subChapters: Seq[Chapter]) {

}

case class SurveyTemplate(id:UUID,
                          name: String,
                          version: Int,
                          authorName: String,
                          preDefinedChapter: Seq[Chapter]/*, dummy: Option[String] = None*/) {

}

class SurveyTemplateDAO {

  private val ending = ".pst"
  private val storageDirectory = new File("C:/Users/André Schmidt/Desktop/tmp")

  def toFileName(x: SurveyTemplate): String = toFileName(x.id)
  def toFileName(x: UUID): String = s"$x$ending"
  def toFile(x: SurveyTemplate) = new File(storageDirectory.getCanonicalPath+File.separator+toFileName(x))

  private implicit val schemaForChapter = SchemaFor[Chapter]

  def save(input: SurveyTemplate) = {
    val target = toFile(input)
    val output = AvroOutputStream.binary[SurveyTemplate](target)
    output.write(input)
    output.flush()
    output.close()
    input
  }


  def list(name: Option[String] = None,
           version: Option[Int] = None,
           authorName: Option[String] = None,
           drop: Int = 0,
           take: Int = 100) = {

    storageDirectory.listFiles(new FilenameFilter {
      override def accept(dir: File, name: String): Boolean = name.endsWith(ending)
    }).take(1).foreach{e =>
      println("OTHER WAY")
      val i = AvroInputStream.binary[SurveyTemplate](e)
      println("BINARY " + i)
      println(i.records.size())
      //println( Try(i.iterator().toList.head) )
      val p = Try(i.iterator().toList.head).toOption
      i.close()
      println(p)
      println("OTHER WAY ------------------")
      println("OTHER WAY ------------------")
      println("OTHER WAY ------------------")
    }


    storageDirectory.listFiles(new FilenameFilter {
      override def accept(dir: File, name: String): Boolean = name.endsWith(ending)
    }).map{e =>
      println("FOUND " + e.getName)
      //      val schema = SchemaBuilder.builder()
      //      val r = Schema.createRecord(java.util.Arrays.asList())
      //      val t = new AvroBinaryInputStream[SurveyTemplate](new SeekableFileInput(e),readerSchema = Some(r))
      //      println(t)

      val i = AvroInputStream.binary[SurveyTemplate](e,AvroSchema[SurveyTemplate]/*,AvroSchema[OldSurveyTemplate]*/)
      println("BINARY " + i)
      println(i.records.size())
      //println( Try(i.iterator().toList.head) )
      val p = Try(i.iterator().toList.head).toOption
      i.close()
      println(p)
      println("------------------")
      println("ENDE")
      p
      //      if( authorName.map(_.equalsIgnoreCase(p.authorName)).getOrElse( name.map(_.equalsIgnoreCase(p.name) ).getOrElse( version.forall(_ == p.version) ) ) ){
      //        Some(p)
      //      } else None
    }.filter(_.isDefined).slice(drop,take).map(_.get)
  }

}

我的sbt：

name := "avroTest"

version := "0.1"

scalaVersion := "2.12.4"

libraryDependencies ++= Seq(
  "com.sksamuel.avro4s" %% "avro4s-core" % "1.8.1"
)

Answer 1

为了使SurveyTemplate向后兼容，我们必须确保新的读取器可以从旧的写入器读取数据。这意味着新参数必须有一个默认值，因为旧的编写器不能提供它。这意味着：

case class SurveyTemplate(id:UUID,
                          name: String,
                          version: Int,
                          preDefinedChapter: Seq[Chapter],
                          dummyString: String = “”)

或者让dummyString成为Option[String]。

另一件事是，您不能更改参数的顺序，除非您为序列化程序提供了编写器模式。

Answer 2

更改AvroInputStream.binary tp AvroInputStream.data允许我添加进化！这不是很好，因为我的文件的二进制表示更适合我的任务。

但是在二进制中，所有的测试都失败了！