Ouestion: Send only file details

[thecontinium]

Hi

I am dealing with large files and don't want to push them all through Kafka but I do want to send the file details when they change of are new; name etc.

Can this connector just send the file details alone in one Kafka message rather than the contents ?

Cheers

[fhussonnois]

Hi @thecontinium, currently the File Pulse has no reader for doing that but this is an easy to add feature. So for example, for each source file, the connector will send one message like this :

{
  "name":"kafka-connect.log",
  "path":"/var/log/kafka",
  "size":172559,
  "lastModified":1560772525000,
  "inode":1705406,
  "hash":661976312
}

Is that what you want ?

[fhussonnois]

Hi @thecontinium, I've just release File Pulse 1.4.1 which include the new reader FileInputMetadataReader that you can configure for achieving what illustrated in my previous message.

[thecontinium]

Great! I’ll give that a go. Thanks for following up Florian. On 12 August 2020 at 16:48:27, Florian Hussonnois (notifications@github.com<mailto:notifications@github.com>) wrote: Hi @thecontinium<https://github.com/thecontinium>, I've just release File Pulse 1.4.1 which include the new reader FileInputMetadataReader<https://streamthoughts.github.io/kafka-connect-file-pulse/docs/developer-guide/file-readers/#fileinputmetadatareader> that you can configure for achieving what illustrated in my previous message. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#57 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACHXEBSKGBYMC5JRASER3KDSAK2URANCNFSM4PYVIP5A>.

[fhussonnois]

Hi could you please close this issue if your need is resolved. Thanks.

[thecontinium]

Resolved. Thanks.