OutOfMemoryError on tomcat7

Question

我正在开发一个web应用程序，它接受一个zip文件，由用户上传，在服务器上解压缩，并处理这些文件。如果压缩文件不太大(20-25MB)，但是如果文件大约或超过(50 It )，它就会产生OutOfMemoryError。

我试图通过将export CATALINA_OPTS="-Xmx1024M"添加到startup.sh中的tomcat7来增加java最大内存分配池，但错误仍然存在。

AFAIK，问题在于解压缩.zip文件。top显示，tomcat在提取50 of文件时使用800 of的内存。是否有任何解决方案，以支持高达200 to的上传，同时有效地使用可用的内存？

解压缩代码如下：

package user;

import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;

public class unzip {

public void unzipFile(String filePath, String oPath)
{

    FileInputStream fis = null;
    ZipInputStream zipIs = null;
    ZipEntry zEntry = null;
    try {
        fis = new FileInputStream(filePath);
        zipIs = new ZipInputStream(new BufferedInputStream(fis));
        while((zEntry = zipIs.getNextEntry()) != null){
            try{
                byte[] tmp = new byte[8*1024];
                FileOutputStream fos = null;
                String opFilePath = oPath+zEntry.getName();
                System.out.println("Extracting file to "+opFilePath);
                fos = new FileOutputStream(opFilePath);
                int size = 0;
                while((size = zipIs.read(tmp)) != -1){
                    fos.write(tmp, 0 , size);
                }
                fos.flush();
                fos.close();
            }catch(Exception ex){

            }
        }
        zipIs.close();
        fis.close();
    } catch (FileNotFoundException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}
}

错误代码如下：

HTTP Status 500 - javax.servlet.ServletException:      java.lang.OutOfMemoryError: Java heap space

type Exception report

message javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space

description The server encountered an internal error that prevented it from fulfilling this request.

exception

org.apache.jasper.JasperException: javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
    org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:549)
    org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:455)
    org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

root cause

javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
    org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:916)
    org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:845)
    org.apache.jsp.Upload_jsp._jspService(Upload_jsp.java:369)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

root cause

java.lang.OutOfMemoryError: Java heap space
    org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:322)
    org.apache.commons.io.output.DeferredFileOutputStream.getData(DeferredFileOutputStream.java:213)
    org.apache.commons.fileupload.disk.DiskFileItem.getSize(DiskFileItem.java:289)
org.apache.jsp.Upload_jsp._jspService(Upload_jsp.java:159)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

note The full stack trace of the root cause is available in the Apache Tomcat/7.0.52 (Ubuntu) logs.
Apache Tomcat/7.0.52 (Ubuntu)

令人惊讶的是，catalina.out文件中没有关于此异常的任何内容。

提前谢谢。

在Upload.jsp中编辑DiskFileItem的代码

//necessary imports go here
File file ;
int maxFileSize = 1000 * 1000 * 1024;
int maxMemSize = 1000 * 1024;
ServletContext context = pageContext.getServletContext();
String filePath = context.getInitParameter("file-upload");
String contentType = request.getContentType();
if(contentType != null)
{
  if ((contentType.indexOf("multipart/form-data") >= 0)) 
  {
  DiskFileItemFactory factory = new DiskFileItemFactory();
  factory.setSizeThreshold(maxMemSize);
  factory.setRepository(new File("/tmp/"));
  ServletFileUpload upload = new ServletFileUpload(factory);
  upload.setSizeMax( maxFileSize );
  try{ 
     List fileItems = upload.parseRequest(request);
     Iterator i = fileItems.iterator();
     while (i.hasNext ()) 
     {

        FileItem fi = (FileItem)i.next();
        if ( !fi.isFormField () )   
        {
           String fieldName = fi.getFieldName();
           String fileName = fi.getName();
           if(fileName.endsWith(".zip")||fileName.endsWith(".pdf")||fileName.endsWith(".doc")||fileName.endsWith(".docx")||fileName.endsWith(".ppt")||fileName.endsWith(".pptx")||fileName.endsWith(".html")||fileName.endsWith(".htm")||fileName.endsWith(".epub")||fileName.endsWith(".djvu"))
           {
              boolean isInMemory = fi.isInMemory();
              long sizeInBytes = fi.getSize();            
              new File(filePath+fileName).mkdir();
              filePath = filePath+fileName+"/";
              file = new File( filePath + fileName.substring( fileName.lastIndexOf("/"))) ;
              fi.write(file);
              String fileExtension = FilenameUtils.getExtension(fileName);
              if(fileExtension.equals("zip"))
              {
                 System.out.println("In zip.");
                 unzip mfe = new unzip();
                 mfe.unzipFile(filePath+fileName,filePath);
                 File zip = new File(filePath+fileName);
                 zip.delete();
              }
              File corePath = new File(filePath);
              int count=0;
           //some more processing
           }
        }
     }
  }
  catch(Exception e)
  {
     //exception handling goes here      
}
  }
}

Answer 1

问题不在您发布的解压缩代码中。根目录位于：

java.lang.OutOfMemoryError: Java heap space
    org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:322)
    org.apache.commons.io.output.DeferredFileOutputStream.getData(DeferredFileOutputStream.java:213)
    org.apache.commons.fileupload.disk.DiskFileItem.getSize(DiskFileItem.java:289)

你注意到ByteArrayOutputStream.toByteArray了吗？因此，您似乎是在给增长过快的ByteArrayOutputStream写信。请找到并发布使用此ByteArrayOutputStream的代码，因为您的邮政编码不使用这种东西。

更新:从您发布的代码中获得，您的代码似乎没有问题。但是FileItem.getSize()电话做了一些讨厌的事情：

283   public long getSize() {
284        if (size >= 0) {
285            return size;
286        } else if (cachedContent != null) {
287            return cachedContent.length;
288        } else if (dfos.isInMemory()) {
289            return dfos.getData().length;
290        } else {
291            return dfos.getFile().length();
292        }
293    }

如果文件项的数据存储在内存中，则调用getData()，后者调用toByteArray()

209    public byte[]  [More ...] getData()
210    {
211        if (memoryOutputStream != null)
212        {
213            return memoryOutputStream.toByteArray();
214        }
215        return null;
216    }

然后分配一个新的数组：

317    public synchronized byte[] toByteArray() {
318        int remaining = count;
319        if (remaining == 0) {
320            return EMPTY_BYTE_ARRAY; 
321        }
322        byte newbuf[] = new byte[remaining];
           //Do stuff
333        return newbuf;
334    }

因此，在很短的时间内，你有两倍的正常内存消耗。

我建议你：

1. 将maxMemSize设置为无更多8-32 Kb。
2. 为JVM进程提供更多内存:例如，-Xmx2g
3. 确保您没有保存对FileItem的任何不必要的引用，因为在您当前的配置中，它们消耗了大量内存。
4. 如果OOM再次发生，采取堆堆。您可以使用-XX:+HeapDumpOnOutOfMemoryError JVM标志自动为您创建堆转储。然后，您可以使用堆转储分析器(例如，Eclipse )来检查谁在分配这么多内存以及在哪里分配内存。

Answer 2

问题是，当用户上传zip文件时，整个zip文件在内存中读取，在堆栈跟踪中，在调用

DiskFileItem.getSize()

从DiskFileItem的源代码中，DiskFileItem.getSize()首先获取所有数据，

public long getSize() {
284        if (size >= 0) {
285            return size;
286        } else if (cachedContent != null) {
287            return cachedContent.length;
288        } else if (dfos.isInMemory()) {
289            return dfos.getData().length;
290        } else {
291            return dfos.getFile().length();
292        }
293    }

通过查看DeferredFileOutputStream.getDate()的文档

Returns either the output file specified in the constructor or the temporary file created or null.
If the constructor specifying the file is used then it returns that same output file, even when threashold has not been reached.
If constructor specifying a temporary file prefix/suffix is used then the temporary file created once the threashold is reached is returned If the threshold was not reached then null is returned.

Returns:
    The file for this output stream, or null if no such file exists.

理想的用户不应该被允许上传任何大小的文件，应该有一个最大的大小限制给您的服务器容量。

Answer 3

为每个zip条目分配8MB似乎只是空中方法中的一个指点。尝试使用较小的缓冲区，例如不超过1kb。垃圾收集不能持续进行。

尝试使用以下方法：

int BUFFER_SIZE = 1024;
int size;
byte[] buffer = new byte[BUFFER_SIZE];

...
FileOutputStream out = new FileOutputStream(path, false);
BufferedOutputStream fout = new BufferedOutputStream(out, BUFFER_SIZE);

while ( (size = zin.read(buffer, 0, BUFFER_SIZE)) != -1 ) {
   fout.write(buffer, 0, size);
}