Tim Spann

This is the code cleaned up for rag injupyter

Source Code

https://github.com/tspannhw/AIM-NYCStreetCams/tree/main/MultipleVectorsAdvanced%20SearchDataModelDesign

Example run demo in Youtube.

video

https://www.youtube.com/watch?v=HaRc0rsaMo0

Any updates to this since KRaFT?

https://github.com/tspannhw/pulsar-thermal-pinot/blob/main/weather.md

Put them into 2 docker nodes. Are you using https://hub.docker.com/r/apache/nifi-registry Is configuration right? It has to store data.https://nifi.apache.org/docs/nifi-registry-docs/html/getting-started.html

Upgrade to 1.9. Is it kerberized or have a login?

See https://community.hortonworks.com/articles/227560/real-time-stock-processing-with-apache-nifi-and-ap.html

Source: https://community.hortonworks.com/storage/attachments/93299-stock-to-kafka.xml https://community.hortonworks.com/storage/attachments/93298-stocks-copy.json https://github.com/tspannhw/stocks-nifi-kafka

Install Java 8

Install NiFi https://nifi.apache.org/download.html

Install Kafka https://kafka.apache.org/downloads

Or get a linux box or big VM

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.0/index.html

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/index.html

we don't have one to list all topics. you could call kafka-topics.sh to get the list or make an API call

https://community.hortonworks.com/articles/57262/integrating-apache-nifi-and-apache-kafka.html

https://community.hortonworks.com/articles/155527/ingesting-golden-gate-records-from-apache-kafka-an.html

ConsumeKafkaRecord_1_0 (comma list of all topics) to ConvertRecord to PutHDFS you may add ConvertAvroToOrc or PutParquet

it has worked for me. post here https://community.hortonworks.com/gallery/index.html https://community.hortonworks.com/questions/1629/nifi-connection-to-mssql-server-db.html https://community.hortonworks.com/articles/87632/ingesting-sql-server-tables-into-hive-via-apache-n.html

https://github.com/bazaarvoice/jolt/issues/130

I use default values. https://community.hortonworks.com/articles/149910/handling-hl7-records-part-1-hl7-ingest.html

good point, this was a follow up to the other article, should have had a review. sorry.

Josh Long is great, when I worked at Pivotal I got a few articles in the Spring Weekly list.

Spring for Hadoop hasn't been updated in forever. It is stuck on HDP 2.2 and we are on HDP 2.6. Spring Data JDBC and Spring Data Repositories make a lot of sense. I should do that, I'll do an update when I get the chance. Maybe add Java 9 and some other goodies. Thanks for the suggestions. If you want to fork the github repo, please do!

http://opennlp.sourceforge.net/models-1.5/

You need to install the OpenNLP models and reference that in the processor properties. Also OpenNLP misses a lot of names and locations. Accuracy is kind of hit or miss. https://github.com/tspannhw/nifi-nlp-processor https://community.hortonworks.com/articles/163776/parsing-any-document-with-apache-nifi-15-with-apac.html

Jean-Paul,

There's another open source registry that integrates with Kafka and other systems extremely well and has a great REST API and UI:

https://github.com/hortonworks/registry

It has versioning and is moving to adding protocol buffers and going into Apache.

Have you tried that one?

NiFi can run on a cluster of servers to distribute the load. NiFi generally supports 50 megabytes a second per node

have you seen the open source Superset from airbnb and hortonworks

paused on that one will try this weekend

Some ways to do Java 8. https://dzone.com/articles/zlwell-written-java

i have seen a lot of messy code. You bring it into IntelliJ / Eclipse and format the code, hide the bad comments and run some static code analysis tools. Java is nice for that.

This article was from 2013. Those are good options too. This should probably be upgraded.

This article was from 2013. Those are good options too. This should probably be upgraded.

This article was from 2013. Those are good options too. This should probably be upgraded.

This article was from 2013. Those are good options too. This should probably be upgraded.

Have you seen Spring Data REST ?

What about Gemfire XD?

http://www.pivotal.io/big-data/pivotal-gemfire-xd

What about Gemfire?

http://www.pivotal.io/big-data/pivotal-gemfire

Good catch. I like your second idea better though.

Good catch. I like your second idea better though.

Ours was dying on hotmail and gmail only. Turned out to be a mail server issue with one particular email address used for sending. Black list security issue.

/**
*
* @param subject
* @param message
* @param emailAddress
* @return
*/
public final static String sendEmail(String subject, String message, String emailAddress) {
// email sent
boolean emailSent = false;

// email message
StringBuilder emailLoggingMessage =
new StringBuilder(UserProvisioningUtility.gkBUFFER_SIZE);

// start of message
emailLoggingMessage
.append(Messages.getString("UserProvisioningProcess.SMTP_MESSAGE_LOG"))
.append(emailAddress).append(System.getProperty("line.separator"));

Email email = new SimpleEmail();
//email.setHostName("smtp.googlemail.com");
//email.setSmtpPort(465);
email.setHostName(Messages.getString("UserProvisioningProcess.EMAIL_SERVER"));

int smtpPort = 25;
if ( null != Messages.getString("UserProvisioningProcess.EMAIL_PORT")) {
try {
smtpPort = Integer.parseInt(Messages.getString("UserProvisioningProcess.EMAIL_PORT"));
}
catch(Throwable t) {
emailLoggingMessage.append(t.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
}
email.setSmtpPort(smtpPort);

// email.setAuthenticator(new DefaultAuthenticator("username", "password"));
// email.setSSLOnConnect(true);
try {
email.setFrom(Messages.getString("UserProvisioningProcess.FROM_EMAIL"));
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
email.setSubject(subject);
try {
email.setMsg(message);
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
try {
email.addTo(emailAddress);
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
try {
email.send();
emailSent = true;
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}

// sent message
if (emailSent) {
emailLoggingMessage.append(
Messages.getString("UserProvisioningProcess.SMTP_MESSAGE_SENT"))
.append(System.getProperty("line.separator"));
}

return emailLoggingMessage.toString();
}