本人尝试用了JXL, POI3.2到3.8版本按照一般的方式来生出大量的数据到Excel,例如100000条数据,基本上都会出现OOM的情况。
下面的例子给我们介绍POI是怎么处理大量数据写到EXCEL里面的:
/* ====================================================================
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
==================================================================== */
package org.apache.poi.xssf.usermodel.examples;
import java.io.*;
import java.util.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipOutputStream;
import org.apache.poi.ss.usermodel.DateUtil;
import org.apache.poi.ss.usermodel.IndexedColors;
import org.apache.poi.ss.util.CellReference;
import org.apache.poi.xssf.usermodel.*;
/**
* Demonstrates a workaround you can use to generate large workbooks and avoid OutOfMemory exception.
*
* The trick is as follows:
* 1. create a template workbook, create sheets and global objects such as cell styles, number formats, etc.
* 2. create an application that streams data in a text file
* 3. Substitute the sheet in the template with the generated data
*
* <p>
* Since 3.8-beta3 POI provides a low-memory footprint SXSSF API which implementing the "BigGridDemo" strategy.
* XSSF is an API-compatible streaming extension of XSSF to be used when
* very large spreadsheets have to be produced, and heap space is limited.
* SXSSF achieves its low memory footprint by limiting access to the rows that
* are within a sliding window, while XSSF gives access to all rows in the
* document. Older rows that are no longer in the window become inaccessible,
* as they are written to the disk.
* </p>
* See <a "http://poi.apache.org/spreadsheet/how-to.html#sxssf">
* http://poi.apache.org/spreadsheet/how-to.html#sxssf</a>.
*
* @author Yegor Kozlov
*/
public class BigGridDemo {
private static final String XML_ENCODING = "UTF-8";
public static void main(String[] args) throws Exception {
// Step 1. Create a template file. Setup sheets and workbook-level objects such as
// cell styles, number formats, etc.
XSSFWorkbook wb = new XSSFWorkbook();
XSSFSheet sheet = wb.createSheet("Big Grid");
Map<String, XSSFCellStyle> styles = createStyles(wb);
//name of the zip entry holding sheet data, e.g. /xl/worksheets/sheet1.xml
String sheetRef = sheet.getPackagePart().getPartName().getName();
//save the template
FileOutputStream os = new FileOutputStream("template.xlsx");
wb.write(os);
os.close();
//Step 2. Generate XML file.
File tmp = File.createTempFile("sheet", ".xml");
Writer fw = new OutputStreamWriter(new FileOutputStream(tmp), XML_ENCODING);
generate(fw, styles);
fw.close();
//Step 3. Substitute the template entry with the generated data
FileOutputStream out = new FileOutputStream("big-grid.xlsx");
substitute(new File("template.xlsx"), tmp, sheetRef.substring(1), out);
out.close();
}
/**
* Create a library of cell styles.
*/
private static Map<String, XSSFCellStyle> createStyles(XSSFWorkbook wb){
Map<String, XSSFCellStyle> styles = new HashMap<String, XSSFCellStyle>();
XSSFDataFormat fmt = wb.createDataFormat();
XSSFCellStyle style1 = wb.createCellStyle();
style1.setAlignment(XSSFCellStyle.ALIGN_RIGHT);
style1.setDataFormat(fmt.getFormat("0.0%"));
styles.put("percent", style1);
XSSFCellStyle style2 = wb.createCellStyle();
style2.setAlignment(XSSFCellStyle.ALIGN_CENTER);
style2.setDataFormat(fmt.getFormat("0.0X"));
styles.put("coeff", style2);
XSSFCellStyle style3 = wb.createCellStyle();
style3.setAlignment(XSSFCellStyle.ALIGN_RIGHT);
style3.setDataFormat(fmt.getFormat("$#,##0.00"));
styles.put("currency", style3);
XSSFCellStyle style4 = wb.createCellStyle();
style4.setAlignment(XSSFCellStyle.ALIGN_RIGHT);
style4.setDataFormat(fmt.getFormat("mmm dd"));
styles.put("date", style4);
XSSFCellStyle style5 = wb.createCellStyle();
XSSFFont headerFont = wb.createFont();
headerFont.setBold(true);
style5.setFillForegroundColor(IndexedColors.GREY_25_PERCENT.getIndex());
style5.setFillPattern(XSSFCellStyle.SOLID_FOREGROUND);
style5.setFont(headerFont);
styles.put("header", style5);
return styles;
}
private static void generate(Writer out, Map<String, XSSFCellStyle> styles) throws Exception {
Random rnd = new Random();
Calendar calendar = Calendar.getInstance();
SpreadsheetWriter sw = new SpreadsheetWriter(out);
sw.beginSheet();
//insert header row
sw.insertRow(0);
int styleIndex = styles.get("header").getIndex();
sw.createCell(0, "Title", styleIndex);
sw.createCell(1, "% Change", styleIndex);
sw.createCell(2, "Ratio", styleIndex);
sw.createCell(3, "Expenses", styleIndex);
sw.createCell(4, "Date", styleIndex);
sw.endRow();
//write data rows
for (int rownum = 1; rownum < 100000; rownum++) {
sw.insertRow(rownum);
sw.createCell(0, "Hello, " + rownum + "!");
sw.createCell(1, (double)rnd.nextInt(100)/100, styles.get("percent").getIndex());
sw.createCell(2, (double)rnd.nextInt(10)/10, styles.get("coeff").getIndex());
sw.createCell(3, rnd.nextInt(10000), styles.get("currency").getIndex());
sw.createCell(4, calendar, styles.get("date").getIndex());
sw.endRow();
calendar.roll(Calendar.DAY_OF_YEAR, 1);
}
sw.endSheet();
}
/**
*
* @param zipfile the template file
* @param tmpfile the XML file with the sheet data
* @param entry the name of the sheet entry to substitute, e.g. xl/worksheets/sheet1.xml
* @param out the stream to write the result to
*/
private static void substitute(File zipfile, File tmpfile, String entry, OutputStream out) throws IOException {
ZipFile zip = new ZipFile(zipfile);
ZipOutputStream zos = new ZipOutputStream(out);
@SuppressWarnings("unchecked")
Enumeration<ZipEntry> en = (Enumeration<ZipEntry>) zip.entries();
while (en.hasMoreElements()) {
ZipEntry ze = en.nextElement();
if(!ze.getName().equals(entry)){
zos.putNextEntry(new ZipEntry(ze.getName()));
InputStream is = zip.getInputStream(ze);
copyStream(is, zos);
is.close();
}
}
zos.putNextEntry(new ZipEntry(entry));
InputStream is = new FileInputStream(tmpfile);
copyStream(is, zos);
is.close();
zos.close();
}
private static void copyStream(InputStream in, OutputStream out) throws IOException {
byte[] chunk = new byte[1024];
int count;
while ((count = in.read(chunk)) >=0 ) {
out.write(chunk,0,count);
}
}
/**
* Writes spreadsheet data in a Writer.
* (YK: in future it may evolve in a full-featured API for streaming data in Excel)
*/
public static class SpreadsheetWriter {
private final Writer _out;
private int _rownum;
public SpreadsheetWriter(Writer out){
_out = out;
}
public void beginSheet() throws IOException {
_out.write("<?xml version=\"1.0\" encoding=\""+XML_ENCODING+"\"?>" +
"<worksheet xmlns=\"http://schemas.openxmlformats.org/spreadsheetml/2006/main\">" );
_out.write("<sheetData>\n");
}
public void endSheet() throws IOException {
_out.write("</sheetData>");
_out.write("</worksheet>");
}
/**
* Insert a new row
*
* @param rownum 0-based row number
*/
public void insertRow(int rownum) throws IOException {
_out.write("<row r=\""+(rownum+1)+"\">\n");
this._rownum = rownum;
}
/**
* Insert row end marker
*/
public void endRow() throws IOException {
_out.write("</row>\n");
}
public void createCell(int columnIndex, String value, int styleIndex) throws IOException {
String ref = new CellReference(_rownum, columnIndex).formatAsString();
_out.write("<c r=\""+ref+"\" t=\"inlineStr\"");
if(styleIndex != -1) _out.write(" s=\""+styleIndex+"\"");
_out.write(">");
_out.write("<is><t>"+value+"</t></is>");
_out.write("</c>");
}
public void createCell(int columnIndex, String value) throws IOException {
createCell(columnIndex, value, -1);
}
public void createCell(int columnIndex, double value, int styleIndex) throws IOException {
String ref = new CellReference(_rownum, columnIndex).formatAsString();
_out.write("<c r=\""+ref+"\" t=\"n\"");
if(styleIndex != -1) _out.write(" s=\""+styleIndex+"\"");
_out.write(">");
_out.write("<v>"+value+"</v>");
_out.write("</c>");
}
public void createCell(int columnIndex, double value) throws IOException {
createCell(columnIndex, value, -1);
}
public void createCell(int columnIndex, Calendar value, int styleIndex) throws IOException {
createCell(columnIndex, DateUtil.getExcelDate(value, false), styleIndex);
}
}
}
分享到:
相关推荐
java poi导出大量数据到Excel
Java解析大数据量Excel,支持解析百万行excel数据,十万数据基本上十来秒就解析完毕,亲测过最大excel1048576行数据用时219秒,硬件好点的PC估计跑起来更快。
由于项目需要对大量Excel数据进行输入输出处理,在使用JXL,POI后发现很容易出现OOM,最后在网上找到阿里的开源...经过大量的调试优化,现通过JAVA生成104万行20列的数据并写入到Excel文件的Sheet中只需要70秒的时间。
java 使用 poi 解析导入大数据量(几万数据量+)时,报出OOM。这是使用POI 第二种处理方法,解决大数据量导入内存溢出问题,并提升效率
java中将数据和图片导出到Excel文件
轻松解决普通poi形式导出Excel的中出现的栈溢出问题,此资源可实现千万级数据分批导出csv文件,csv大数据量导出(千万级别,不会内存溢出),多线程导出 ,生产环境已经很稳定的使用着
根据在数据库配置查询的字段,筛选字段,用户可以在在前台面精确配置自己要查询的字段,同时该系统支持查询大量的数据,同时用户可以根据自己的配置多大文件对Execel分文件下载。
主要为大家详细介绍了java导出大批量即百万以上数据的excel文件,具有一定的参考价值,感兴趣的小伙伴们可以参考一下
用开源 Apache POI 技术导出Excel,解决导出大数据出现OOM、栈溢出问题,此资源可实现百万级数据多线程分批导出Excel文件,不会内存溢出,生产环境已很稳定的使用者,所以用到的技术很核心、值得参考
该资源主要实现如下功能,利用poi获取excel中大量数据,然后利用jdbc批量插入到mysql
主要为大家详细介绍了Java实现excel大数据量导入,文中示例代码介绍的非常详细,具有一定的参考价值,感兴趣的小伙伴们可以参考一下
开发测试时候导入数据遇到大数据导入的问题,整理了下,需要的朋友可以参考下
今天小编就为大家分享一篇解决Java导入excel大量数据出现内存溢出的问题,具有很好的参考价值,希望对大家有所帮助。一起跟随小编过来看看吧
从Excel中读取数据导入到数据库中,Excel中有合并单元格的列,也有超过4000字符的数据列。
poi读取大量数据会造成gc内存溢出的报错,由于垃圾回收机制无法将大量的对象及时的回收,而这些对象又会保存在内存中,会导致内存不够用的情况,这时候我们就需要使用新的方法,读取为cvs即可.此解决方案可支持千万数据的...
poi 方式导入 大量excel数据,将excelDir目录下文件考入f盘即可测试 poi 方式导入 大量excel数据,将excelDir目录下文件考入f盘即可测试
轻松解决普通poi形式导出Excel的中出现的栈溢出问题,此资源可实现千万级数据分批导出csv文件,测试实现16500000条数据大概80秒左右;具体表里内容。
前言 最近有个项目在生产环境做数据导入时,发现开始执行导入任务会出现cpu狂飙的情况。几番定位查找发现是在...这种方式POI会把文件的所有内容都加载到内存中,读取大的excel文件时很容易占用大量内存导致oom的发生
主要为大家详细介绍了Java实现Excel批量导入数据,文中示例代码介绍的非常详细,具有一定的参考价值,感兴趣的小伙伴们可以参考一下
在数据管理系统中,添加大量数据,如果通过手动录入的话,相当费事费力。最近开发一款数据管理系统,所以我花了半天时间将该功能梳理出来。该资料将框架和其他功能都简化,主要实现该功能。