novatechflow

Searching and filtering large IP address datasets within PostgreSQL can be challenging. Why? Databases excel at data storage and structured queries, but often struggle with full-text search and complex analysis. Apache Solr, a high-performance search engine built on top of Lucene, is designed to handle these tasks with remarkable speed and flexibility. What do we need? A running PostgreSQL database with a table containing IP address information (named "ip_loc" in our example). A basic installation of Apache Solr. Setting up Apache Solr 1. Create a Solr Core: solr create -c ip_data -d /path/to/solr/configsets/ 2. Define the Schema ( schema.xml ) <field name="start_ip" type="ip" indexed="true" stored="true"/> <field name="end_ip" type="ip" indexed="true" stored="true"/> <field name="iso2" type="string" indexed="true" stored="true"/> <field...

Since the most interesting Apache NiFi parts are coming from ASF [1] or Hortonworks [2], I thought to use CDH 5.7 and do the same, just to be curious. Here's now my 30 minutes playground, currently running in Googles Compute. On one of my playground nodes I installed Apache NiFi per mkdir /software && cd /software &&  wget http://mirror.23media.de/apache/nifi/0.6.1/nifi-0.6.1-bin.tar.gz   && tar xvfz nifi-0.6.1-bin.tar.gz Then I've set only nifi.sensitive.props.key property in conf/nifi.properties to an easy to remember secret. The next bash /software/nifi-0.6.1/bin/nifi.sh install installs Apache NiFi as an service. After log in into Apache NiFi's WebUI, download and add the template [3] to Apache NiFi, move the template icon to the drawer, open it and edit the twitter credentials to fit your developer account. To use an  schema-less SolR index (or Cloudera Search in CDH) I copied some example files over into a local directory: cp -r ...