ftp 相关问题

在python上ftp下载当天的最新文件

下午好。 请帮我解决问题。需要用python连接到ftp服务器并从五个不同的文件夹中下载今天才出现的所有文件。删除 fi...

错误：没有这样的文件（sftp liximomo 扩展名）

当我使用 Visual Studio 代码的扩展名 liximomo/vscode-sftp 在服务器上上传文件时遇到此问题。 [错误]错误：没有这样的文件 在 SFTPStream._transform 在 SFTPStream。

扭曲的DTPFactory超时

我正在尝试使用twisted设置一个简单的FTP服务器： sudo venv/bin/twistd ftp -p 21 -r /home/ftpuser/ftproot/ --auth 文件:/home/ftpuser/pass.dat 当我从 Mac 上运行这个程序时，它运行得很好。 ...

Apache Camel - 将文件写入 ftp 失败

下面是我的简单文件组件消费者到 ftp 生产者组件的路由。 来自（“文件：//C：//文件文件夹”） .to("ftp://ftp01.test.com/myfolder/?用户名=user1234&密码=pass1234&延迟=5s&

使用 FtpWebRequest 上传时出现“请求的 URI 无效”

我尝试将文件上传到 FTP 服务器上的目录。我将此方法与 FtpWebRequest 一起使用。 我想将一个文件上传到该用户的主目录，但我总是收到以下错误...

如果我想将 .pgp 和 .zip 文件从 FTP 重复传输到 Azure Blob 存储，这是最佳选择

如果我想将 .pgp 和 .zip 文件从 FTP 重复传输到 Azure Blob 存储，最好的选择是什么？ 我们有多个 FTP 配置，它们存储在 Snowflake 数据库中。我们可以

Python 脚本通过 FTP 上传文件

我想制作一个脚本来上传文件到FTP。 登录系统如何工作？我正在寻找这样的东西： ftp.login=(mylogin) ftp.pass=(我的密码) 以及任何其他登录凭据。

在 FTPWebrequest 中启用 SSL 后无法连接 FileZilla

我尝试在 ftpwebrequest 中使用 SSL，如下所示： FileStream outputStream = new FileStream(fileName, FileMode.Append); reqFTP = (FtpWebRequest)FtpWebRequest.Create(new Uri("ft...

如何将文件解压缩到 .NET 内存流？

我有文件（来自第 3 方）正在通过 FTP 传输到我们服务器上的目录。我下载它们并处理它们甚至“x”分钟。效果很好。 现在，某些文件是 .zip 文件。这意味着...

Laravel 存储文件上传时损坏

所以我一直在我的项目中使用ftp函数（手动设置$conn_id，制作fpt_put（$conn_id，...），conn_close等）， 现在我在控制器中添加了“使用存储”，设置主机、用户名和密码...

SSHJ SFTPException：超出当前上下文的最大并发传输数

我不断得到 SFTPException：超出当前上下文的最大并发传输数 处理文件时 我正在使用 SSHJ v0.38.0 和 Spring Boot 3.0.3 的最新依赖项 我不断得到 SFTPException：超出当前上下文的最大并发传输数 处理文件时 我正在使用 SSHJ v0.38.0 与 Spring Boot 3.0.3 的最新依赖项 <dependency> <groupId>com.hierynomus</groupId> <artifactId>sshj</artifactId> <version>0.38.0</version> </dependency> 下面是我的完整错误和异常日志 2024-06-05T14:32:34.858+08:00 | INFO | Now checking /043/Output/FILE//FILE2024-05-30-08-20-22.txt - com.vendor.robbijournal.service.SftpService.processFiles:154 [scheduling-1:16216] 2024-06-05T14:32:34.858+08:00 | DEBUG | Now read the content of file FILE2024-05-30-08-20-22.txt | current 6 / remaining 6 / total 12 - com.vendor.robbijournal.service.SftpService.processFiles:157 [scheduling-1:16216] 2024-06-05T14:32:34.859+08:00 | DEBUG | Opening `/043/Output/FILE/FILE2024-05-30-08-20-22.txt` - net.schmizz.sshj.sftp.SFTPClient.open:75 [scheduling-1:16216] 2024-06-05T14:32:35.097+08:00 | DEBUG | Sending close - net.schmizz.sshj.connection.channel.direct.SessionChannel.sendClose:287 [scheduling-1:16216] 2024-06-05T14:32:35.336+08:00 | DEBUG | Got chan request for `exit-status` - net.schmizz.sshj.connection.channel.direct.SessionChannel.gotChannelRequest:334 [sshj-Reader-xfer.robbi.com/127.0.0.1:22-1717568411096:16216] 2024-06-05T14:32:35.337+08:00 | DEBUG | Got close - net.schmizz.sshj.connection.channel.direct.SessionChannel.gotClose:220 [sshj-Reader-xfer.robbi.com/127.0.0.1:22-1717568411096:16216] 2024-06-05T14:32:35.339+08:00 | DEBUG | Forgetting `session` channel (#2) - net.schmizz.sshj.connection.ConnectionImpl.forget:89 [sshj-Reader-xfer.robbi.com/127.0.0.1:22-1717568411096:16216] 2024-06-05T14:32:35.340+08:00 | ERROR | We caught error : Maximum concurrent transfers exceeded for the current context - com.vendor.robbijournal.service.SftpService.fetchAndStoreFiles:137 [scheduling-1:16216] net.schmizz.sshj.sftp.SFTPException: Maximum concurrent transfers exceeded for the current context at net.schmizz.sshj.sftp.Response.error(Response.java:140) at net.schmizz.sshj.sftp.Response.ensurePacketTypeIs(Response.java:117) at net.schmizz.sshj.sftp.SFTPEngine.open(SFTPEngine.java:169) at net.schmizz.sshj.sftp.SFTPClient.open(SFTPClient.java:76) at net.schmizz.sshj.sftp.SFTPClient.open(SFTPClient.java:81) at net.schmizz.sshj.sftp.SFTPClient.open(SFTPClient.java:86) at com.vendor.robbijournal.service.SftpService.processFiles(SftpService.java:158) at com.vendor.robbijournal.service.SftpService.fetchAndStoreFiles(SftpService.java:118) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:354) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:392) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720) at com.vendor.robbijournal.service.SftpService$$SpringCGLIB$$0.fetchAndStoreFiles(<generated>) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130) at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124) at io.micrometer.observation.Observation.observe(Observation.java:499) at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:124) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) 以下是我的服务课程 package com.vendor.robbijournal.service; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.InputStream; import java.nio.charset.StandardCharsets; import java.sql.Timestamp; import java.util.List; import java.util.Optional; import org.apache.commons.codec.digest.DigestUtils; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Service; import com.vendor.robbijournal.constant.WhichXferFileStatus; import com.vendor.robbijournal.constant.WhichXferFileType; import com.vendor.robbijournal.model.Flag; import com.vendor.robbijournal.model.Sol; import com.vendor.robbijournal.model.XferFtpFiles; import com.vendor.robbijournal.repository.FlagRepository; import com.vendor.robbijournal.repository.XferFtpFilesRepository; import com.vendor.robbijournal.utils.FilePermissionUtil; import com.vendor.robbijournal.utils.LoggerFactoryUtil; import ch.qos.logback.classic.Logger; import jakarta.transaction.Transactional; import net.schmizz.sshj.SSHClient; import net.schmizz.sshj.sftp.FileAttributes; import net.schmizz.sshj.sftp.RemoteResourceInfo; import net.schmizz.sshj.sftp.SFTPClient; import net.schmizz.sshj.transport.verification.PromiscuousVerifier; @Service @Transactional public class SftpService { private static final Logger LOG = LoggerFactoryUtil.getLogger(); @Autowired private SSHClient sshClient; @Autowired private XferFtpFilesRepository repository; @Autowired private FlagRepository flagRepository; @Value("${spring.profiles.active}") private String activeProfile; @Value("${spring.jpa.properties.hibernate.jdbc.time_zone}") private String hibernateTimeZone; @Value("${sftp.host}") private String host; @Value("${sftp.port}") private int port; @Value("${sftp.username}") private String username; @Value("${sftp.password}") private String password; private String[] DIRECTORIES = { "/043/Output/FILE/" }; @Scheduled(fixedRate = 60000) // Run every 60 seconds //@Scheduled(fixedRate = 600000) // run every hour public void fetchAndStoreFiles() throws IOException { LOG.trace("Starting to fetch files from SFTP server.."); try { // find Flag record SFTP_FETCHING List<Flag> flag = flagRepository.findByKey("SFTP_FETCHING"); if (flag.size() == 0) { Flag f = new Flag(); f.setKey("SFTP_FETCHING"); f.setValue1("ON"); f.setTimeZone(hibernateTimeZone); flagRepository.save(f); } if (flag.size() > 0) { Flag f = flag.get(0); if(f.getValue1().equalsIgnoreCase("ON")) { LOG.trace("SFTP_FETCHING in progress..abort this process"); return; // Abort the current execution } f.setValue1("ON"); flagRepository.save(f); } // Check if FTP session is active LOG.debug("sshClient.isConnected() = " + sshClient.isConnected()); if (!sshClient.isConnected()) { LOG.debug("Look like not connected to SFTP server.. let reconnecting.."); sshClient = new SSHClient(); sshClient.addHostKeyVerifier(new PromiscuousVerifier()); // Configure your SSH connection details here sshClient.connect(host, port); // KeyProvider keys = sshClient.loadKeys("path/to/private/key", "keyPassword"); // sshClient.authPublickey("username", keys); LOG.debug("Connected to SFTP server: " + host + ":" + port + "using username: " + username+ " and password: " + password); sshClient.authPassword(username, password); } LOG.debug("sshClient.isAuthenticated() = " + sshClient.isAuthenticated()); try (SFTPClient sftpClient = sshClient.newSFTPClient()) { for (String directory : DIRECTORIES) { processFiles(sftpClient, directory); } LOG.debug("Finish checked {} directories!",DIRECTORIES.length); sftpClient.close(); LOG.debug("sftpClient.close()"); } sshClient.disconnect(); LOG.debug("Done fetching file and sshClient.isConnected() = " + sshClient.isConnected() + "! "); // find Flag record flag = flagRepository.findByKey("SFTP_FETCHING"); if (flag.size() > 0) { Flag f = flag.get(0); f.setValue1("OFF"); flagRepository.save(f); } } catch (Exception e) { LOG.error("We caught error : {}", e.getMessage(), e); List<Flag> flag = flagRepository.findByKey("SFTP_FETCHING"); flag = flagRepository.findByKey("SFTP_FETCHING"); if (flag.size() > 0) { Flag f = flag.get(0); f.setValue1("OFF"); flagRepository.save(f); } e.printStackTrace(); } } private void processFiles(SFTPClient sftpClient, String directory) throws IOException { LOG.info("Let check {}", directory); List<RemoteResourceInfo> files = sftpClient.ls(directory); int fileNumber = 0; for (RemoteResourceInfo entry : files) { LOG.info("Now checking {}/{}", directory, entry.getName()); if (!entry.isDirectory()) { fileNumber = fileNumber+1; LOG.debug("Now read the content of file {} | current {} / remaining {} / total {}", entry.getName(), fileNumber, files.size() - fileNumber, files.size()); try (InputStream inputStream = sftpClient.open(entry.getPath()).new RemoteFileInputStream()) { ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); /* * 1024 byte = 1K * 8192 byte = 8K * 16384 byte = 16K */ byte[] buffer = new byte[16384]; int readCount; while ((readCount = inputStream.read(buffer)) > 0) { byteArrayOutputStream.write(buffer, 0, readCount); } String fileContent = byteArrayOutputStream.toString(StandardCharsets.UTF_8); // Count the number of lines in the file content long lineCount = fileContent.lines().count(); String md5sum = DigestUtils.md5Hex(byteArrayOutputStream.toByteArray()); // Check if a file with the same MD5 already exists Optional<XferFtpFiles> existingFile = repository.findByMd5sum(md5sum); if (existingFile.isPresent()) { if (entry.getName().equalsIgnoreCase(existingFile.get().getFilename())) { LOG.warn("File {} with the same MD5 {} already exists, now skip this file", entry.getName(), md5sum); } else { LOG.warn("File {} with MD5 {} have same content as {}, now skip this file", entry.getName(), md5sum, existingFile.get().getFilename()); } continue; } // Logic to determine version : Check if a file with the same filename already // exists List<XferFtpFiles> existingFiles = repository.findByFilename(entry.getName()); Integer newVersion = 1; if (!existingFiles.isEmpty()) { LOG.debug("Seem we have {} record with same file name {}", existingFiles.size(), entry.getName()); List<XferFtpFiles> maxVersionOptList = repository .findByFilenameOrderByVersionAsc(entry.getName()); for (XferFtpFiles maxVersionOpt : maxVersionOptList) { newVersion = maxVersionOpt.getVersion(); maxVersionOpt.setStatus(WhichXferFileStatus.IGNORED_USE_NEW_VERSION); repository.save(maxVersionOpt); // find Sol and update the status also List<Sol> solList = maxVersionOpt.getSol(); for (Sol sol : solList) { // sol.setStatus(WhichXferFileStatus.IGNORED_USE_NEW_VERSION); // repository.save(sol); } } newVersion = newVersion + 1; LOG.warn("Seems {} have an update, set new update as version {}", entry.getName(), newVersion); } FileAttributes attrs = entry.getAttributes(); String permissions = FilePermissionUtil.getPermissionString(attrs); XferFtpFiles xferFtpFile = new XferFtpFiles(); xferFtpFile.setHost(sshClient.getRemoteHostname()); xferFtpFile.setPort(Integer.toString(sshClient.getRemotePort())); xferFtpFile.setDirectory(directory); xferFtpFile.setPermission(permissions); xferFtpFile.setFilename(entry.getName()); xferFtpFile.setPayload(fileContent); xferFtpFile.setPayloadLine((int) lineCount); xferFtpFile.setMd5sum(md5sum); xferFtpFile.setVersion(newVersion); xferFtpFile.setTs_ftp(new Timestamp((long) attrs.getMtime() * 1000)); xferFtpFile.setTimeZone(hibernateTimeZone); xferFtpFile.setStatus(WhichXferFileStatus.NEW); xferFtpFile.setFileType(WhichXferFileType.TO_CHECK); xferFtpFile.setFilesize(attrs.getSize()); // Save the file to DB repository.save(xferFtpFile); } } } } } 这是 SSHClient 的 bean package com.vendor.robbijournal.config; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import net.schmizz.sshj.SSHClient; @Configuration public class SshClientConfig { @Bean public SSHClient sshClient() { SSHClient sshClient = new SSHClient(); // Additional configuration if needed return sshClient; } } package com.vendor.robbijournal.config; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import net.schmizz.sshj.SSHClient; @Configuration public class SshClientConfig { @Bean public SSHClient sshClient() { SSHClient sshClient = new SSHClient(); // Additional configuration if needed return sshClient; } } 我不知道sshj（实际上也不知道Java），但是检查sshj源代码，它看起来不像RemoteFileOutputStream.close实际上关闭了远程文件。 只是 RemoteFile.close 关闭它。 因此您需要确保流和文件都已关闭： try ( RemoteFile remoteFile = sftpClient.open(entry.getPath()); InputStream inputStream = remoteFile.new RemoteFileInputStream() ) {

使用 SSH.NET 连接到端口 990 上的 FTP 站点时“服务器响应包含空字符”

我正在尝试使用 Renci SSH.NET 连接到 FTP 站点并从我的 C# 应用程序下载文件。我可以使用 FileZilla 和 WinSCP（在 Windows 10 Pro 64 上）成功连接到主机，并通过...

SSHJ - SFTPException：超出当前上下文的最大并发传输数

我不断收到 SFTPException：超出当前上下文的最大并发传输数 处理文件时 我正在使用 sshj v0.38.0 和 springboot 3.0.3 的最新依赖项 我不断得到SFTPException: Maximum concurrent transfers exceeded for the current context 处理文件时 我正在使用 sshj v0.38.0 和 springboot 3.0.3 的最新依赖项 <dependency> <groupId>com.hierynomus</groupId> <artifactId>sshj</artifactId> <version>0.38.0</version> </dependency> 下面是我的完整错误和异常日志 2024-06-05T14:32:34.858+08:00 | INFO | Now checking /043/Output/FILE//FILE2024-05-30-08-20-22.txt - com.vendor.robbijournal.service.SftpService.processFiles:154 [scheduling-1:16216] 2024-06-05T14:32:34.858+08:00 | DEBUG | Now read the content of file FILE2024-05-30-08-20-22.txt | current 6 / remaining 6 / total 12 - com.vendor.robbijournal.service.SftpService.processFiles:157 [scheduling-1:16216] 2024-06-05T14:32:34.859+08:00 | DEBUG | Opening `/043/Output/FILE/FILE2024-05-30-08-20-22.txt` - net.schmizz.sshj.sftp.SFTPClient.open:75 [scheduling-1:16216] 2024-06-05T14:32:35.097+08:00 | DEBUG | Sending close - net.schmizz.sshj.connection.channel.direct.SessionChannel.sendClose:287 [scheduling-1:16216] 2024-06-05T14:32:35.336+08:00 | DEBUG | Got chan request for `exit-status` - net.schmizz.sshj.connection.channel.direct.SessionChannel.gotChannelRequest:334 [sshj-Reader-xfer.robbi.com/127.0.0.1:22-1717568411096:16216] 2024-06-05T14:32:35.337+08:00 | DEBUG | Got close - net.schmizz.sshj.connection.channel.direct.SessionChannel.gotClose:220 [sshj-Reader-xfer.robbi.com/127.0.0.1:22-1717568411096:16216] 2024-06-05T14:32:35.339+08:00 | DEBUG | Forgetting `session` channel (#2) - net.schmizz.sshj.connection.ConnectionImpl.forget:89 [sshj-Reader-xfer.robbi.com/127.0.0.1:22-1717568411096:16216] 2024-06-05T14:32:35.340+08:00 | ERROR | We caught error : Maximum concurrent transfers exceeded for the current context - com.vendor.robbijournal.service.SftpService.fetchAndStoreFiles:137 [scheduling-1:16216] net.schmizz.sshj.sftp.SFTPException: Maximum concurrent transfers exceeded for the current context at net.schmizz.sshj.sftp.Response.error(Response.java:140) at net.schmizz.sshj.sftp.Response.ensurePacketTypeIs(Response.java:117) at net.schmizz.sshj.sftp.SFTPEngine.open(SFTPEngine.java:169) at net.schmizz.sshj.sftp.SFTPClient.open(SFTPClient.java:76) at net.schmizz.sshj.sftp.SFTPClient.open(SFTPClient.java:81) at net.schmizz.sshj.sftp.SFTPClient.open(SFTPClient.java:86) at com.vendor.robbijournal.service.SftpService.processFiles(SftpService.java:158) at com.vendor.robbijournal.service.SftpService.fetchAndStoreFiles(SftpService.java:118) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:354) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:392) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720) at com.vendor.robbijournal.service.SftpService$$SpringCGLIB$$0.fetchAndStoreFiles(<generated>) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130) at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124) at io.micrometer.observation.Observation.observe(Observation.java:499) at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:124) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) 以下是我的服务课程 package com.vendor.robbijournal.service; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.InputStream; import java.nio.charset.StandardCharsets; import java.sql.Timestamp; import java.util.List; import java.util.Optional; import org.apache.commons.codec.digest.DigestUtils; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Service; import com.vendor.robbijournal.constant.WhichXferFileStatus; import com.vendor.robbijournal.constant.WhichXferFileType; import com.vendor.robbijournal.model.Flag; import com.vendor.robbijournal.model.Sol; import com.vendor.robbijournal.model.XferFtpFiles; import com.vendor.robbijournal.repository.FlagRepository; import com.vendor.robbijournal.repository.XferFtpFilesRepository; import com.vendor.robbijournal.utils.FilePermissionUtil; import com.vendor.robbijournal.utils.LoggerFactoryUtil; import ch.qos.logback.classic.Logger; import jakarta.transaction.Transactional; import net.schmizz.sshj.SSHClient; import net.schmizz.sshj.sftp.FileAttributes; import net.schmizz.sshj.sftp.RemoteResourceInfo; import net.schmizz.sshj.sftp.SFTPClient; import net.schmizz.sshj.transport.verification.PromiscuousVerifier; @Service @Transactional public class SftpService { private static final Logger LOG = LoggerFactoryUtil.getLogger(); @Autowired private SSHClient sshClient; @Autowired private XferFtpFilesRepository repository; @Autowired private FlagRepository flagRepository; @Value("${spring.profiles.active}") private String activeProfile; @Value("${spring.jpa.properties.hibernate.jdbc.time_zone}") private String hibernateTimeZone; @Value("${sftp.host}") private String host; @Value("${sftp.port}") private int port; @Value("${sftp.username}") private String username; @Value("${sftp.password}") private String password; private String[] DIRECTORIES = { "/043/Output/FILE/" }; @Scheduled(fixedRate = 60000) // Run every 60 seconds //@Scheduled(fixedRate = 600000) // run every hour public void fetchAndStoreFiles() throws IOException { LOG.trace("Starting to fetch files from SFTP server.."); try { // find Flag record SFTP_FETCHING List<Flag> flag = flagRepository.findByKey("SFTP_FETCHING"); if (flag.size() == 0) { Flag f = new Flag(); f.setKey("SFTP_FETCHING"); f.setValue1("ON"); f.setTimeZone(hibernateTimeZone); flagRepository.save(f); } if (flag.size() > 0) { Flag f = flag.get(0); if(f.getValue1().equalsIgnoreCase("ON")) { LOG.trace("SFTP_FETCHING in progress..abort this process"); return; // Abort the current execution } f.setValue1("ON"); flagRepository.save(f); } // Check if FTP session is active LOG.debug("sshClient.isConnected() = " + sshClient.isConnected()); if (!sshClient.isConnected()) { LOG.debug("Look like not connected to SFTP server.. let reconnecting.."); sshClient = new SSHClient(); sshClient.addHostKeyVerifier(new PromiscuousVerifier()); // Configure your SSH connection details here sshClient.connect(host, port); // KeyProvider keys = sshClient.loadKeys("path/to/private/key", "keyPassword"); // sshClient.authPublickey("username", keys); LOG.debug("Connected to SFTP server: " + host + ":" + port + "using username: " + username+ " and password: " + password); sshClient.authPassword(username, password); } LOG.debug("sshClient.isAuthenticated() = " + sshClient.isAuthenticated()); try (SFTPClient sftpClient = sshClient.newSFTPClient()) { for (String directory : DIRECTORIES) { processFiles(sftpClient, directory); } LOG.debug("Finish checked {} directories!",DIRECTORIES.length); sftpClient.close(); LOG.debug("sftpClient.close()"); } sshClient.disconnect(); LOG.debug("Done fetching file and sshClient.isConnected() = " + sshClient.isConnected() + "! "); // find Flag record flag = flagRepository.findByKey("SFTP_FETCHING"); if (flag.size() > 0) { Flag f = flag.get(0); f.setValue1("OFF"); flagRepository.save(f); } } catch (Exception e) { LOG.error("We caught error : {}", e.getMessage(), e); List<Flag> flag = flagRepository.findByKey("SFTP_FETCHING"); flag = flagRepository.findByKey("SFTP_FETCHING"); if (flag.size() > 0) { Flag f = flag.get(0); f.setValue1("OFF"); flagRepository.save(f); } e.printStackTrace(); } } private void processFiles(SFTPClient sftpClient, String directory) throws IOException { LOG.info("Let check {}", directory); List<RemoteResourceInfo> files = sftpClient.ls(directory); int fileNumber = 0; for (RemoteResourceInfo entry : files) { LOG.info("Now checking {}/{}", directory, entry.getName()); if (!entry.isDirectory()) { fileNumber = fileNumber+1; LOG.debug("Now read the content of file {} | current {} / remaining {} / total {}", entry.getName(), fileNumber, files.size() - fileNumber, files.size()); try (InputStream inputStream = sftpClient.open(entry.getPath()).new RemoteFileInputStream()) { ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); /* * 1024 byte = 1K * 8192 byte = 8K * 16384 byte = 16K */ byte[] buffer = new byte[16384]; int readCount; while ((readCount = inputStream.read(buffer)) > 0) { byteArrayOutputStream.write(buffer, 0, readCount); } String fileContent = byteArrayOutputStream.toString(StandardCharsets.UTF_8); // Count the number of lines in the file content long lineCount = fileContent.lines().count(); String md5sum = DigestUtils.md5Hex(byteArrayOutputStream.toByteArray()); // Check if a file with the same MD5 already exists Optional<XferFtpFiles> existingFile = repository.findByMd5sum(md5sum); if (existingFile.isPresent()) { if (entry.getName().equalsIgnoreCase(existingFile.get().getFilename())) { LOG.warn("File {} with the same MD5 {} already exists, now skip this file", entry.getName(), md5sum); } else { LOG.warn("File {} with MD5 {} have same content as {}, now skip this file", entry.getName(), md5sum, existingFile.get().getFilename()); } continue; } // Logic to determine version : Check if a file with the same filename already // exists List<XferFtpFiles> existingFiles = repository.findByFilename(entry.getName()); Integer newVersion = 1; if (!existingFiles.isEmpty()) { LOG.debug("Seem we have {} record with same file name {}", existingFiles.size(), entry.getName()); List<XferFtpFiles> maxVersionOptList = repository .findByFilenameOrderByVersionAsc(entry.getName()); for (XferFtpFiles maxVersionOpt : maxVersionOptList) { newVersion = maxVersionOpt.getVersion(); maxVersionOpt.setStatus(WhichXferFileStatus.IGNORED_USE_NEW_VERSION); repository.save(maxVersionOpt); // find Sol and update the status also List<Sol> solList = maxVersionOpt.getSol(); for (Sol sol : solList) { // sol.setStatus(WhichXferFileStatus.IGNORED_USE_NEW_VERSION); // repository.save(sol); } } newVersion = newVersion + 1; LOG.warn("Seems {} have an update, set new update as version {}", entry.getName(), newVersion); } FileAttributes attrs = entry.getAttributes(); String permissions = FilePermissionUtil.getPermissionString(attrs); XferFtpFiles xferFtpFile = new XferFtpFiles(); xferFtpFile.setHost(sshClient.getRemoteHostname()); xferFtpFile.setPort(Integer.toString(sshClient.getRemotePort())); xferFtpFile.setDirectory(directory); xferFtpFile.setPermission(permissions); xferFtpFile.setFilename(entry.getName()); xferFtpFile.setPayload(fileContent); xferFtpFile.setPayloadLine((int) lineCount); xferFtpFile.setMd5sum(md5sum); xferFtpFile.setVersion(newVersion); xferFtpFile.setTs_ftp(new Timestamp((long) attrs.getMtime() * 1000)); xferFtpFile.setTimeZone(hibernateTimeZone); xferFtpFile.setStatus(WhichXferFileStatus.NEW); xferFtpFile.setFileType(WhichXferFileType.TO_CHECK); xferFtpFile.setFilesize(attrs.getSize()); // Save the file to DB repository.save(xferFtpFile); } } } } } 这是 SSHClient 的 bean package com.vendor.robbijournal.config; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import net.schmizz.sshj.SSHClient; @Configuration public class SshClientConfig { @Bean public SSHClient sshClient() { SSHClient sshClient = new SSHClient(); // Additional configuration if needed return sshClient; } } package com.vendor.robbijournal.config; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import net.schmizz.sshj.SSHClient; @Configuration public class SshClientConfig { @Bean public SSHClient sshClient() { SSHClient sshClient = new SSHClient(); // Additional configuration if needed return sshClient; } } 我不知道sshj（实际上也不知道Java），但是检查sshj源代码，它看起来不像RemoteFileOutputStream.close实际上关闭了远程文件。 只是 RemoteFile.close 关闭它。 因此您需要确保流和文件都已关闭： try ( RemoteFile remoteFile = sftpClient.open(entry.getPath()); InputStream inputStream = remoteFile.new RemoteFileInputStream() ) {

Filezilla FTP 服务器 - 客户端更改目录失败

我在我的电脑上设置了 Filezilla FTP 服务器并尝试从本地计算机本身进行 ftp。我可以连接、传输文件，但无法更改远程目录。 默认情况下，它位于主目录...

跟踪文件是否已通过 php 正确上传到 ftp 上

我需要跟踪，如果文件已成功上传，则将值（1或0）存储到mysql数据库中。 我想要的是确保用户上传了文件并且传输正常，所以没有呃...

如何在 git-ftp 客户端上使用 sftp？

我正在尝试使用 jenkins 来自动执行 sftp 上传和替换，以便 sftp 服务器上的文件夹等于 git 存储库。这是使用托管在运行 nginx 的 openshift 上的 sftp，仅

如何使用 php 从 ftp 复制以相同字符串开头的所有文件

如何使用 php 从 ftp 复制以相同字符串开头的所有文件？我可以使用以下代码复制一个文件： if(copy('ftp://testuser:pass@ftpserver/file_20240520170720.MIS','file.MIS')) { ...

Whonix 上的 Dh 密钥太小（Python FTP）

我正在尝试从 Whonix 操作系统内将文件上传到服务器。我能够在 Whonix 中使用 Filezilla 成功完成此操作，所以我不太确定为什么 Python 代码不成功

通过 FTP 将课程上传到 Moodle

我有一个非常大的单个课程的 Moodle .mbz 备份文件 - 略高于 1GB。我需要将此备份上传到服务器，但是当然 Moodle 提供的文件管理器不允许我这样做...

无法通过 ngrok 使用 filezilla 连接到 ftp 服务器

我正在使用 ngrok 公开公开 ftp 服务器。当我使用 file zilla 客户端连接到 ftp 服务器时，它显示错误“服务器发送了带有不可路由地址的被动回复。我们...