Gtjkyu

Could I set Flink time window to a large value? Could I set DataStream time window to a large value like 24 hours? The reason for the requirement is that I want to make data statistics based on the latest 24 hours client traffic to the web site. This way, I can check if there are security violations. For example, check if a user account used multiple source IPs to log on to the web site. Or check how many unique pages a certain IP accessed in the latest 24 hours. If security violation is detected, the configured action will be taken in real time such as blocking the source IP or locking the relevant user account. The throughput of the web site is around 200Mb/s. I think setting the time window to a large value will cause memory issue. Should I store the statistics results of each time window like 5 minutes into database? Then make statistics based on database query for the date generated in the latest 24 hours? I don't have any experience with big data analysis. Any advice will be ...

Update Postgres Using Flink My dataset named dbData contains set of data. I want to update postgres with that data regularly where the dbData changes regularly on everyday purpose. dbData.map(new Write()) .output(JDBCOutputFormat.buildJDBCOutputFormat() .setDrivername(Utils.properties_fetch("drivername")) .setDBUrl(Utils.properties_fetch("dbURL")) .setUsername(Utils.properties_fetch("username")) .setPassword(Utils.properties_fetch("password")) .setQuery( Write.updatequery) .finish()); My "Write" class looks like the following: public class Write implements MapFunction<Tuple7<String, String, String, String, String, String, String>, Row> { static String updatequery ; private static final long serialVersionUID = 1L; public Row map( Tuple7<String, String, String, String, String, String, String> value)throws Exception { Row obj = new Row(7); ...

Trouble with deserializing Avro data in Scala I am building an Apache Flink application in Scala which reads streaming data from a Kafka bus and then performs summarizing operations on it. The data from Kafka is in Avro format and needs a special Deserialization class. I found this scala class AvroDeserializationScehema (http://codegists.com/snippet/scala/avrodeserializationschemascala_saveveltri_scala): package org.myorg.quickstart import org.apache.avro.io.BinaryDecoder import org.apache.avro.io.DatumReader import org.apache.avro.io.DecoderFactory import org.apache.avro.reflect.ReflectDatumReader import org.apache.avro.specific.{SpecificDatumReader, SpecificRecordBase} import org.apache.flink.api.common.typeinfo.TypeInformation import org.apache.flink.api.java.typeutils.TypeExtractor import org.apache.flink.api.common.serialization._ import java.io.IOException class AvroDeserializationSchema[T](val avroType: Class[T]) extends DeserializationSchema[T] { private var reader: DatumRead...